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Foreword 


Quality assurance for higher education in Europe has developed significantly since 
2002, and has increasingly influenced, and been influenced by, the Bologna Process. 

A major step in the Bologna Process was taken at the ministerial meeting in Bergen 
in May 2005, with the adoption of the Standards and Guidelines for Quality Assurance 
in the European Higher Education Area (ESG) and of the Framework for Qualifications 
of the European Higher Education Area (FQEHEA). The ESG were developed to 
meet the need for a common understanding of quality assurance in European higher 
education. The FQEHEA was developed to meet the need for a common understanding 
of the generic learning outcomes that signified completion of each of the three cycles of 
higher education. Both documents have encouraged change and are having a growing 
impact on quality assurance procedures. 

The present report updates the findings of the first ENQA survey on Quality 
Procedures in European Higher Education, undertaken by the Danish Evaluation 
Institute (EVA) in 2002, and considers the main developments that have taken place 
since then. In addition, the survey looks at agencies' views on their compliance with 
Part 3 of the ESG; their attitudes towards the European Quality Assurance Register for 
Higher Education (EQAR); and their external review plans. 

ENQA is pleased to present the 2008 Quality Procedures Project report. The results 
of this survey offer a valuable insight into quality assurance procedures across Europe, 
and I hope this survey will contribute to further future discussions between agencies. 


Peter Williams, 

President 

European Association for Quality Assurance in Higher Education (ENQA) 


Glossary 

An external quality procedure could be one of the following: 

• An evaluation of a subject 1 2 , which focuses on the quality of one specific subject, 
typically in all the programmes in which this subject is taught. 

• An evaluation of a programme, which focuses on the activities within a study 
programme, which in this context is defined as studies leading to a formal degree. 

• An evaluation of an institution, which examines the quality of all activities 
within an institution, i.e. organisation, financial matters, management, facilities, 
teaching and research. 

• An evaluation of a theme which examines the quality or practice of a specific 
theme within education e.g. ICT or student counselling. 

• An audit, which is an evaluation of the strengths and weaknesses of the quality 
mechanisms established by an institution itself to continuously monitor and 
improve the activities and services of either a subject, a programme, the whole 
institution or a theme. 

• An accreditation process, which builds on the same methodological elements 
as the other types of evaluation, but differs from the other procedures in that 
judgement is provided according to predefined standards to decide whether a 
given subject, programme, institution or theme meets the necessary level. 

• Benchmarking, which is a comparison of results between subjects, programmes, 
institutions or themes leading to an exchange of experiences of best practice. 

• Criteria are seen as checkpoints and benchmarks for assessing the quality of the 
input and the process. 

• Standards are seen as the expected outcomes of the educational training. For 
example standards defined by professional organisation or legislation. It concerns 
the competencies that are expected from the graduates. 

Readers should note that aforementioned definitions are valid for this project 
only. 


1 In this survey the term ‘external quality procedure’ covers the terms ‘evaluation’, ‘assessment’, ‘review’, ‘audit’, etc. 

2 A subject is for example the subject ‘chemistry’ within the study programme of medicine. 


Executive Summary 

This paper is concerned with the 2008 Quality Procedures survey on quality assurance 
agencies within the European Higher Education Area (EHEA) and beyond. Undertaken 
by the European Association for Quality Assurance in Higher Education (ENQA), it 
collates and analyses responses from 51 agencies and related bodies, and provides an 
update on a similar survey undertaken in 2002. 

The initial chapter identifies the responding agencies, their countries and their 
ENQA status, and the extent of their remits. Most but not all agencies work within a 
national remit but a few also have international ones as well. Remits vary from those 
covering only one type of institution to those with a far wider range of responsibilities, 
often encompassing essentially all higher education provision, and in some cases 
various support services as well. The ‘level’ at which quality assurance responsibilities 
are held shows some interesting patterns; in about half of the cases the institutions are 
responsible for approval of new subject areas, but when it comes to programmes the 
role of the relevant QA agency rises markedly. Approval of new institutions is typically 
done by central or regional government, but not in all cases. 

Chapter 2 provides details on the status, structures, activities and resources of 
the QA agencies. It pays particular attention to those issues covered by Part 3 of the 
European Standards and Guidelines (ESG), and is of interest regarding criteria that are 
also used for ENQA membership. The analysis and commentary acknowledge that there 
can be a strained relationship between national traditions, legislation and the ESG. 

The responses illustrate that almost all agencies are formally recognised, either under 
a specific law (most) or by public authorities. The number of agencies within a country 
varies; in most cases there is a single agency, but regional and other dimensions can 
result in there being multiple agencies within a country. The analysis of the types of 
activities covered by agencies shows that, whilst two-thirds of agencies use programme 
level procedures, just less than half work at institutional level. Some agencies cover 
both. QA procedures are generally undertaken on a cyclical rather than one-off basis, 
and generally result in formal consequences. One of the significant sets of responses 
in this part of the survey indicates that, within a five year period, three-quarters of 
the agencies have undertaken or are undertaking change, or planning changes to their 
procedures. Data on staffing, funding and other resources provide an interesting snap- 
shot of generally adequate provision, but also illustrate some constraints under which 
agencies are working. 

The third and most extensive chapter is concerned with structure^), functions, 
decision making and methodological approaches to external quality assurance. A 
very rich source of data illustrates the increasing extent to which stakeholders are 
now involved within agency structures, although the main functions of agencies 
(external quality assurance and improvement, and dissemination of information 
about quality) remain much the same as in 2003. Agencies have however taken on, or 
become involved in, a wider range of additional functions, particularly in relation to 
the development of the Bologna Process. One interesting finding is that the four-stage 
theoretical process of quality assurance, presented in an annex of the ESG, is applied 
by most but not all of the responding agencies. Within the details of what is examined 
during the external evaluations, research strategies and assessment ranked highly, but 


a surprising 20% of agencies indicated that they do not include institutions’ assessments 
of students in their evaluations. A variety of mechanisms is used by agencies to look at 
their own internal quality assurance procedures with, as might be expected, the more 
developed agencies having a wider range of measures than those established more 
recently. 

Chapter 4 provides further evidence that builds upon important aspects coming 
from Chapter 3. It describes and discusses a number of critical areas relating to the 
division of responsibilities within the methodological frameworks and procedures used 
by agencies. It reports on: the membership of external panels [there is clear evidence 
of an increase in student membership but a (corresponding?) decrease in the direct 
involvement of agency staff on panels)]; the training of panels; the responsibilities 
exercised in determining what QA method(s) will be applied (mostly the agency’s) and 
the writing of reports (mostly the panel’s). Legal regulations provide a reference frame 
for many agencies in determining their methodologies, but there are other important 
factors as well. An interesting comparison with the 2003 survey may be drawn from the 
now wider availability and use of ‘specific criteria and standards’ for QA. The quality 
assurance of collaborative provision between (providing) institutions (including trans- 
national delivery) was not covered in 2003; the data provided here may act as a baseline 
for future comparisons in an area that is becoming of increasing importance. 

Chapter 5 focuses on just one critical aspect of QA - site visits, and the preparatory 
work undertaken before them. Evidence suggests that the development of the self- 
evaluation reports is done by institutions in much the same way as in 2003; sources for 
data collection are much as might be expected and the people and panels interviewed 
much as in 2003. The site visits themselves are generally, but not always, accompanied 
by agency staff, and whilst averages can be distorting it was perhaps interesting to 
see that site visits for institutional evaluations are only one day longer than those for 
programme evaluations. Class room observations were used in a small proportion of 
methodologies / cases only. 

The sixth chapter covers the extent to which, and the ways in which, the outcomes 
of the QA procedures are reported. Most agencies provide public reports on their 
evaluations, with those that do not yet do so are planning to. The majority of reports, 
which include data analyses, recommendations, and conclusions, are subject to 
some form of ‘consultation’ with the institution before final publication. It is usually 
the agency that published the report, with the number of reports published clearly 
depending on the size of an agency’s ‘jurisdiction’ and whether it is involved in only 
institutional evaluation or programme evaluations, when the number of reports rises 
considerably, or both. 

This is followed in Chapter 7, as in any typical QA methodology, by consideration 
of follow-up procedures after any judgements and recommendations have been made. 
Various responsibilities lie with both agencies and institutions, and there are a range of 
actions that can be asked for or required of institutions. 

Some preliminary thoughts about possible future developments are covered briefly 
in the penultimate chapter. Revisions to external QA procedures figure highly in the 
responses, as does the development of national qualifications frameworks. Almost 


all agencies have undergone, are undergoing, or are planning to undergo an external 
evaluation themselves, although a small proportion do not, as yet, intend to apply for 
membership of the European Quality Assurance Register (EQAR). 

The final chapter aims to summarise the key findings not in a repetitive way but by 
seeking to provide interesting insights into similarities and differences through cross 
referring detailed findings and discussion points that have arisen from various parts of 
the survey. 


Introduction 


Background and purpose 

The Ministers of the Bologna Process signatory states, in the Prague Communique 
of 19 May 2001, ‘called upon the universities and other higher education institutions, 
national agencies and the European Network of Quality Assurance in Higher Education 
(ENQA) 3 4 , in cooperation with corresponding bodies from countries which are not 
members of ENQA, to collaborate in establishing a common framework of reference 
and to disseminate best practice’ and added that ‘the European University Association, 
the European Association of Institutions in Higher Education (EURASHE), the 
National Unions of Students in Europe and the Council of Europe should be consulted 
in the follow-up work’. 

The first ENQA survey on Quality Procedures in European Higher Education was 
undertaken by the Danish Evaluation Institute (EVA) in 2002 and formed part of the 
response to this mandate. That survey addressed higher education quality assurance 
practices in the countries of ENQA members. The survey findings were published under 
the title ‘Quality Procedures in European Higher Education - An ENQA Survey’ 4 which 
for the first time detailed the evaluation methods used by quality assurance agencies 
in Europe. One reported finding was that major progress had been made towards 
convergence in basic methods and procedures despite national agencies having varying 
priorities 5 . 

Given the significant developments of European quality assurance that have 
occurred since 2002 - including the adoption of the Standards and Guidelines for 
Quality Assurance in the European Higher Education Area (ESG) and the Framework for 
Qualifications of the European Higher Education Area (FQEHEA) - the Board of ENQA 
decided in May 2007 that it should update, using project funding from the European 
Commission, the 2002 findings by conducting a new survey of ENQA members, 
affiliates and associates. For this purpose, ENQA launched the second project on 
“Quality Procedures in European Higher Education”. 

The project goals were: 

1. to update the 2002 survey; 

2. to assess the agencies’ opinion on their compliance with Part 3 of the ESG; 

3. to survey agencies’ external review plans and 

4. to survey agencies’ attitudes towards the European Quality Assurance Register for 
Higher Education (EQAR). 

Process and Method 

The project was divided into three phases. 

In the first phase the Steering and Project groups designed and developed the 
second Quality Procedures questionnaire (attached as an annex in the report) and 
implemented it using the SurveyMonkey electronic tool for web-based questionnaires. 


3 ENQA was transformed from a network into an association in 2004. 

4 European Network for Quality Assurance in Higher Education, Quality Procedures in European Higher Education - An ENQA 
Survey, Occasional Paper No. 5, ISBN 951-98680-8-9, Helsinki, 2003 

5 ENQA statement to the Conference of European Ministers of Education in Berlin, 18-19 September 2003 


The online questionnaire was distributed to the ENQA member agencies, associates 
and affiliates on 21 December 2007 with a response deadline of 31 January 2008. 

The second phase involved preliminary analysis of the responses. For this purpose 
the survey was divided into seven sections namely: (i) Introduction and background 
information about the higher education system (10 questions); (ii) Agencies’ operations 
(14 questions); (iii) Typical methodological approaches to external quality assurance 
(14 questions); (iv) Division of responsibilities and methodological framework for 
procedures involving panels (11 questions); (v) Self-evaluation by the institution/ 
programme (7 questions); (vi) Reporting the outcome of agencies' external quality 
procedures and follow-up (9 questions); (vii) Future developments (6 questions). The 
task of analysing each section and writing the corresponding chapter was assigned to 
specific members of the Project Group, according to their areas of interest. 

Finally, the third phase involved the final analysis and the production of the report. 

Organisation of the project 

The Higher Education and Training Awards Council (HETAC) and the Quality 
Assurance Agency for Higher Education (QAA) took the lead the project. 

A Steering Group and a Project Group were set up to conduct the survey and to draft 
the report. However, in the course of the project the two groups merged into one, as 
it was felt that in this case one group with members of equal tasks and responsibilities 
was more functional. The ENQA Secretariat carried out the administrative 
responsibility of the project. 

The original Steering Group comprised four individuals, i.e. three representatives 
of ENQA member agencies, including an ENQA Board member, plus a member of the 
ENQA Secretariat: 

• Nick Harris, the Quality Assurance Agency in Higher Education (QAA) 

• Peter Cullen, the Higher Education and Training Award Council (HETAC) 

• Achim Hopbach, German Accreditation Council (GAC), ENQA Board Member 

• Emmi Helle, ENQA Secretariat 

The initial Project Group consisted of nine members, i.e. six representatives of ENQA 
member agencies (including a student representative and an ENQA Board member), 
a representative of a higher education institution and two members of the ENQA 
Secretariat. Peter Cullen and Emmi Helle were acting respectively as chairman and 
secretary: 

• Peter Cullen, HETAC 

• Fiona Crozier, QAA 

• Josep Grifoll, the Agency for Quality Assurance in the Catalan University System 
(AQU Catalonia) 

• Helka Kekalainen, the Finnish Higher Education Evaluation Council (FINHEEC) 

• Tanel Sits, the Higher Education Quality Assessment Center of Estonia (HEQAC), 
student representative 

• Kurt Sohm, Fachhochschulrat (Autrian FHR), ENQA Board member 

• Bozana Knezevic, University of Rijeka (Croatia), HEI representative 

• Emmi Helle, ENQA Secretariat 

• Nathalie Costes, ENQA Secretariat 

In the end, the Steering Group and the Project Group acted as one and both, therefore, 
are referred to as the Project Group in this report. 


Each of the Project Group members was responsible for the production of one part of 
this report. The first round of putting together the different chapters and editing them 
was conducted by HETAC (Peter Cullen) and QAA (Nick Harris). ENQA Secretariat 
took care of the final touch, including the layout and publication on the ENQA website. 

Response to the survey 

The survey was largely based on the original survey of 2002, which was updated to 
reflect the relevant developments since then, including the ESG, the FQEHEA, the 
reviews of agencies and the impending establishment of the EQAR. 

The survey was distributed to all ENQA members, affiliates and associates on 21 
December 2007 with a response deadline of 31 January 2008. The findings are based on 
51 responses from agencies from 30 EHEA countries 6 . The Project Group deemed that 
this sample was sufficiently representative of the whole of the EHEA. 

Since 2002, the membership of ENQA has grown steadily and two new categories 
of formal linkage with ENQA were introduced in 2006 namely: association and 
affiliation. Since then 21 entities (networks and organisations) either associated 
or affiliated with ENQA. Therefore, the 2008 survey covered a significantly wider 
distribution than the first, which was completed by 36 quality assurance agencies from 
23 countries. 

Persons completing the survey were given the option to require that their responses 
be reported anonymously in the final report. Nine chose this option. The name and the 
country of those organisations is, therefore, omitted in the text. 

It should be noted that the survey text was the sole channel of communication with 
respondents. Accordingly, the interpretation of questions and criteria may vary, leading 
to potential factual errors or misinterpretations in the results. The authors cannot be 
held responsible for inaccuracies or misunderstanding. Finally, it should also be noted 
that not all questions were completed by all survey respondents. 


6 As of May 2007, 46 countries have signed the Bologna declaration and are thus members of the Bologna Process. 


Chapter 1: 

Introduction and background 
information about higher education 
systems 

Summary information about the 51 respondents is presented in tables 1-2. The 
respondents have been classified according to their relationship with ENQA, namely 
Full member, Candidate member, Associate and Affiliate. The names of the agencies 
in tables 1-2 and in the report are referred to as they were written by those agencies 
which responded to the survey Some agencies are named by their national names 
while other agencies’ names are translated in English. The name and the country of 
those nine organisations that required their responses to be reported anonymously are 
marked with “XX”. 

In the first part of the questionnaire, agencies were asked to state which of the 
following were within their remit or scope: 

i. Universities; 

ii. Higher Education Institutions such as Universities of Applied Science, University 
Colleges, etc; 

iii. A specific discipline or field of learning; 

iv. Other (with a request to specify). 

However, given that the terminology and notion of “universities” and “non-universities” 
varies from country to country, there might have been misunderstandings when 
responding to this question. Respondents could select more than one category. 

1.1 Participating agencies 

Table 1. Full and Candidate members 


NAME OF AGENCY 

COUNTRY 

STATUS 

SCOPE 

Austrian Agency for Quality Assurance (AQA) 

Austria 

Candidate member 

0). 00, (iv) 

National Evaluation and Accreditation Agency (NEAA) 

Bulgaria 

Candidate member 

(i), (ii), (iii), 

(iv) 

XX 

XX 

Candidate member 

(i), (ii) 

National Accreditation Agency (NAA) 

Russia 

Candidate member 

(i), (ii) 

Agency for Quality Assurance in the Galician University 
System (ACSUG) 

Spain 

Candidate member 

(i), (ii) 

XX 

XX 

Candidate member 

(i) 

The European Council on Chiropractic Education (ECCE) 

U.K 

Candidate member 

(ii), (iv) 

Austrian Accreditation Council (AAC) 

Austria 

Full member 

(iv) 

Austrian Fachhochschulrat (FH Council) 

Austria 

Full member 

(ii) 

Council of Flemish Institutions of Higher Education 
(VLHORA) 

Belgium 

Full member 

(ii) 


European University Association - Institutional 
Evaluation Programme (EUA-IEP) 

Belgium 

Full member 


XX 

XX 

Full member 

(i) 

XX 

XX 

Full member 

(ii) 

Accreditation Commission of the Czech Republic 

Czech 

Full member 


Finnish Higher Education Evaluation Council (FINHEEC) 

Finland 

Full member 

(i), (ii) 

Agence devaluation de la Recherche et de l’Enseignement 
Superieur (AERES) 

France 

Full member 

(i), (ii), (iv) 

XX 

XX 

Full member 

(ii), (iv) 

XX 

XX 

Full member 

(i), (ii), (iii) 

Accreditation, Certification and Quality Assurance 
Institute (ACQUIN) 

Germany 

Full member 

0), (ii) 

Central Evaluation and Accreditation Agency Hanover 
(ZEvA) 

Germany 

Full member 

(i), (ii) 

Foundation for International Business Administration 
Accreditation (FIBAA) 

Germany 

Full member 

(i), (ii), (iii), 
(iv) 

German Accreditation Council (GAC) 

Germany 

Full member 

(iv) 

Hungarian Accreditation Committee (HAC) 

Hungary 

Full member 

(i), (ii), (iv) 

Higher Education and Training Awards Council (HETAC) 

Ireland 

Full member 

(ii) 

XX 

XX 

Full member 

(i), (ii) 

Comitato Nazionale Per la Valutazione del Sistema 
Universitario (CNVSU) 

Italy 

Full member 

(i), (ii) 

Higher Education Quality Evaluation Centre (HEQEC) 

Latvia 

Full member 

(i), (ii) 

Accreditation Organisation of the Netherlands and 
Flanders (NVAO) 

Netherlands 

Full member 

(i), (ii) 

Norwegian Agency for Quality Assurance in 
Education (NOKUT) 

Norway 

Full member 

(i), (ii), (iv) 

The Accreditation Commission of Slovak Republic 

Slovakia 

Full member 

(i), (ii) 

Agency for Quality Assurance in the Catalan University 
System (AQU) 

Spain 

Full member 

(i) 

National Agency for Quality Assessment and Accreditation 
(ANECA) 

Spain 

Full member 

(i) 

Swedish National Agency for Higher Education (NAHE) 

Sweden 

Full member 

(i), (ii) 

Center for Accreditation and Quality Assurance of the 
Swiss universities (OAQ) 

Switzerland 

Full member 

(i), (ii) 

The Quality Assurance Agency for Higher 
Education (QAA) 

U.K 

Full member 

(i), (ii), (iv) 


Table 2. Associates and Affiliates 


NAME OF AGENCY 

COUNTRY 

STATUS 

SCOPE 

Public Accreditation Agency for Higher Education 

Albania 

Affiliate 
(CEEN member) 

(ii) 

Central and Eastern European Network of Quality 
Assurance Agencies in Higher Education (CEEN) 

Hungary 

Affiliate 

(iv) 

Center for Quality Assessment in Higher Education 

Lithuania 

Affiliate 
(CEEN member) 

(iv) 

European Consortium for Accreditation (ECA) 

Netherlands 

Affiliate 

(iv) 

Council for Higher Education of the Republic of Slovenia 

Slovenia 

Affiliate 
(CEEN member) 

(ii). (M 

Agency for Science and Higher Education (ASHE) 

Croatia 

Associate 

(i). (ii). (M 

XX 

XX 

Associate 

(i) 

Quality Management Project (HRK) 

Germany 

Associate 

(ii) 

Holy See’s Agency for the Evaluation and Promotion of 
Quality in Ecclesiastical Academic Institutions (AVEPRO) 

Holy See 

Associate 

(i), (ii), (iii) 

Office of Evaluation and Analysis of the Icelandic Ministry 
of Education, Science and Culture in Iceland 

Iceland 

Associate 

(i), (ii), (iv) 

The Council for Higher Education (CHE) 

Israel 

Associate 

(ii) 

European Evangelical Accrediting Association (EEAA) 

Italy 

Associate 

(ii), (iii) 

Higher Education Planning, Evaluation, Accreditation and 
Coordination Council (YODAK) 

North Cyprus 

Associate 

(ii) 

Commission for Accreditation and Quality Assessment 
(CAQA) 

Serbia 

Associate 

(i), (ii) 

XX 

XX 

Associate 

(i) 

Commission for Academic Assessment and Quality 
Improvement at Higher Education (YODEK) 

Turkey 

Associate 

(i), (ii) 


The European system of quality assurance in higher education has expanded through 
the emergence of new agencies since 2002. 

Perusal of tables 1 and 2 and the responses to the survey demonstrate that 
new agencies continue to be established. The Holy See and the governments of 
the Republics of Croatia and Kazakhstan have recently established evaluation (or 
accreditation) agencies responsible for external quality assurance in higher education 7 . 
Greece, Romania and Malta have recently established, or are planning to establish, 
quality assurance agencies 8 . 

In some EU member states, new legislation has led to the restructuring of the higher 
education quality assurance system. In France, for instance, the Comite National 
d’Evaluation (CNE) has been integrated into the new Evaluation Agency for Research 
and Higher Education (AERES), which covers not only all institutions of higher 
education but also research bodies in France. In Denmark, a new act was passed by the 
Parliament in 2007 creating a new accreditation body. 


7 The Agency for Science and Higher Education (ASHE) was established in Croatia in 2004. The National Accreditation Centre of the 
Ministry of Education and Science (NAC) was founded in Kazakhstan in 2005. The Holy See’s Agency for the Evaluation and 
Promotion of Quality in Ecclesiastical Faculties (AVEPRO) was established in 2007. 

8 The Hellenic Quality Assurance Agency for Higher Education (HQAA), established in 2005, is an Associate of ENQA. The 
Romanian Agency for Quality Assurance in Higher Education (ARACIS), established in 2005, is a Candidate member of ENQA. 


1.2 Scope of agencies 

As said in the previous chapter, the agencies were asked to state which of the categories 
- (i), (ii), (iii), (iv) - were within their remit or scope, with the possibility to select more 
than one category. Table 3 illustrates that both ENQA members and associates/affiliates 
(28 agencies covering 24 countries) predominantly evaluate both universities [type 
(i)] and higher education institutions [type (ii)]. It can be deduced that in the majority 
of European higher education quality systems, university (i) and HEI (ii) sectors are 
integrated under one quality framework. 

Table 3 also shows that the remit of the agency is general, in majority of the cases, 
and includes all fields of learning. Only seven respondents specialise in a specific field 
of learning. 

Table 3. 


WHAT IS THE SCOPE OR REMIT OF YOUR AGENCY? 

RESPONSE COUNT 

RESPONSE % 

Both universities (i) and HEI (ii) 

28 

42 

Other (iv) 

16 

24 

Only HEI (ii) 

10 

15 

A specific discipline or field of learning (iii) 

7 

10 

Only universities (i) 

6 

9 

answered question 

50 

skipped question 

1 


The categories specified under “other” included: 

• Evaluation of research institutes, research teams and laboratories; 

• Evaluation of programmes; 

• Evaluation of post secondary vocational programmes and institutions; 

• Evaluation of pre-primary, primary and secondary schools; 

• Other specific disciplines; 

• Private organisations with degree-awarding powers; further education colleges 
offering higher education programmes; 

• Consortium of accreditation organisations; accreditation agencies; 

• Scientific organisations; 

• Engineering schools. 

1.3 The agencies' primary domain 

The responses revealed that most of the quality assurance agencies have been 
established to operate within a country (table 4). Of the seven agencies which operate 
within a part of a territory, four are Spanish regional agencies and two are Belgian 
regional agencies. Among the five organisations which operate within a specific group 
of countries, two are affiliate networks of ENQA: Central and Eastern European 
Network of Quality Assurance Agencies in Higher Education (CEEN) and European 
Consortium for Accreditation (ECA). In addition, four bodies have reported to be 
operating in an international domain: Holy See’s Agency for the Evaluation and 
Promotion of Quality in Ecclesiastical Academic Institutions (AVEPRO), European 
Council on Chiropractic Education (ECCE), EUA’s Institutional Evaluation Programme 
(IEP) and European Evangelical Accrediting Association (EEAA). 


Table 4. 


WHICH OF THE FOLLOWING BEST DESCRIBES YOUR AGENCY'S PRIMARY 
DOMAIN? 

RESPONSE 

COUNT 

RESPONSE % 

A country (i.e. the whole territory of a sovereign state) 

35 

68.6 

A part of the territory of a sovereign state 

7 

13.7 

A specific group of sovereign states 

5 

9.8 

International 

4 

7.8 

Please specify 

13 


answered question 

51 

skipped question 

0 


In table 5, it can be found that, if looking at the numbers, the institutions subject to 
quality procedures of the agencies are in most cases public, type (ii) institutions such as 
university colleges, applied universities, followed by private type (ii) institutions. 

Table 5. 


HOW MANY INSTITUTIONS ARE IN YOUR AGENCY'S 
PRIMARY DOMAIN? 

RESPONSE 

AVERAGE 

RESPONSE 

TOTAL 

RESPONSE 

COUNT 

Public universities 

44 

1543 

35 

Private universities 

29 

958 

33 

Public HEI 

69 

2144 

31 

Private HEI 

52 

1573 

30 

answered question 

45 

skipped question 

6 


In addition, the respondents were asked whether institutions in their agency’s primary 
domain are subject to compulsory external quality procedures by their agency and/or 
others. 42 respondents indicated that all institutions in their agency’s primary domain 
were subject to compulsory external quality procedures against six that indicated the 
opposite. However, it should be noted here that the question’s options were restrictive, 
as respondents had to choose between “yes, all are subject” or “no, none are subject”. 
This may have had a polarising effect in the responses, as the reality is often more 
complicated than just a yes/no-option. For example, AQA from Austria, which had 
chosen the no-answer, stated that public universities (which represent ca 80% of higher 
education) and teacher-training colleges (ca 5%) are recognised by law and obliged to 
implement internal quality management (QM) systems. Evidence of implementation 
is achieved through certification by AQA, but the external quality procedures are not 
compulsory. Fachhochschulen (ca 12%) and private universities (ca 3%), are subject 
to compulsory national accreditation (by the FH Council and the AAC respectively), 
based on evaluation procedures carried out by AQA or other internationally-recognised 
agencies. 

1.4 Approval of new subjects, programmes and institutions 

Table 6 suggests that, in most cases, new subjects are approved by institutions; new 
institutions are approved by central or regional governments and new programmes are 
approved by institutions or QA agencies. Interestingly enough, in five cases, institutions 
are responsible for approving other new institutions. Five agencies skipped this 
question. 
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Table 6. 


WHO IS RESPONSIBLE FOR THE APPROVAL OF NEW SUBJECTS, PROGRAMMES AND 
INSTITUTIONS? 

RESPONSE 

COUNT 


SUBJECTS 

PROGRAMMES 

INSTITUTIONS 

The institutions 

25 

21 

5 

30 

Central government 

9 

16 

21 

26 

Regional government 

5 

11 

16 

16 

The relevant QA agency 

6 

20 

13 

22 

Other 

3 

6 

7 

7 

Please specify 

20 

answered question 

46 

skipped question 

5 


Chapter 2: 

European quality assurance agencies 

2.1 Introduction 

External quality assurance in the European higher education systems has developed 
tremendously in the ten years since the recommendation of the European Council of 24 
September 1998 on European Co-operation in Quality Assurance in Higher Education. 
The developments have been both organisational and structural and have modified the 
role and relevance of external quality assurance. 

The Council recommendation of 1998 highlighted the need for autonomy of the 
agencies and addressed standards for external quality procedures used by agencies. 
Besides the development of quality assurance methodology, one major issue of debate 
since the late nineties has been the question of what standards should be required of 
external quality assurance agencies as organisations. This question was addressed by 
the third part of the “Standards and Guidelines for Quality Assurance in the European 
Higher Education Area” (ESG) which were adopted at the Bologna conference in 2005. 

The ESG comprise a set of standards for agencies that wish to be considered as 
trustworthy and professional in the performance of their functions in the European 
Higher Education Area. The growing importance of these standards as a reference 
frame for agencies is evidenced by the foundation of the European Quality Assurance 
Register (EQAR) on 4 March 2008. 

The ESG were developed to be applicable to all quality assurance agencies in 
Europe (EHEA), irrespective of their structure, function and size and the national 
system in which they operate. However, every agency operates in the context of its 
national higher education system, its assigned role in the national quality assurance 
system, and its national culture and traditions. This means that even though there 
is some convergence in European quality assurance, there are limitations when the 
requirements of ESG conflict with national regulations and traditions. 

This chapter examines the current state of development of ENQA full members, 
candidate members, associates and affiliates, as perceived by the agencies themselves, 
having regard to the national higher education contexts. Questions 11 to 24 of the 
survey paraphrase Part 3 of the ESG and thereby the ENQA membership criteria. 

The analysis and commentary acknowledge that there can be a strained relationship 
between national traditions, legislation and the ESG. 

When interpreting the findings an important underlying question is whether ESG 
are perceived as generic principles which can be satisfied in different ways taking into 
account the national context, or as strict rules to be followed literally. 

It is noteworthy that even though the analysis of this part of the survey follows the 
ESG and thus the membership criteria of ENQA, it does not replace the assessment of 
the fulfilment of the ENQA membership criteria by an agency, nor can any eligibility 
be derived from information contained in this survey. It is the responsibility of the 
ENQA Board to examine the agencies in this respect and decide upon fulfilment of 
membership criteria. 


2.2 Status of the Agency 

Other than specifying autonomy as a necessary characteristic of the agencies the 
Council recommendation of 1998 does not explicitly refer to the status of agencies 
in law. In contrast the ESG stipulate that “agencies should be formally recognised by 
competent public authorities in the EHEA as agencies with responsibilities for external 
quality assurance and should have an established legal basis” 9 . 

This standard seems to be motivated by a principle — one that underpins many 
other parts of the ESG — which holds that the reliability and trustworthiness 
(credibility) of an agency is based on its professionalism in terms of its organisation as 
well as in the performance of its functions. 

In this respect the responses to the question “Does your agency have a legal 
basis?” (table 7) were quite uniform over the agencies irrespective of the relationship 
with ENQA (Full member, Candidate member, Associate and Affiliate). 34 out of 47 
responding agencies answered that they are based on a specific law which is the highest 
possible level of formal recognition. 

Table 7. 


DOES YOUR AGENCY HAVE A LEGAL BASIS? 

RESPONSE COUNT 

RESPONSE % 

Yes 

46 

95.8 

No 

2 

4.2 

Please specify 

28 


answered question 

48 

skipped question 

3 


Ten out of the remaining 13 agencies are recognised by public authorities, just beneath 
this level, and two are not formally recognised at all. Taking into account that four 
of the ten agencies which are recognised by a public authority belong to the German 
accreditation system, and also taking into account that the two agencies that are not 
recognised are international organisations by definition, the survey shows that it is a 
common pattern in the EHEA to establish quality assurance agency(ies) in legislation. 

This picture illustrates the dynamic process of implementing national quality 
assurance systems especially after the Bologna Conference in Berlin in 2003 where 
ministers stated that quality is at the heart of setting up the European Higher Education 
Area and the adoption of the guiding principles of the ESG two years later in Bergen 
2005. 

2.3 Structure of national quality assurance systems and the position of the 
agency 

2.3.1 RESPONSIBILITY OF THE AGENCY 

ESG standard 2.3 stipulates that the agency shall bear formal responsibility for external 
quality assurance. 

The number of agencies in a country is variable. It is influenced by limitations 
on the remit of individual agencies and by the structure of the quality assurance 
system. Evidently, different approaches are feasible. One approach is to have a single 


9 ESG part III, standard 2 


agency that bears sole responsibility for external quality assurance. This can promote 
transparency and simplicity. Alternatively, a country with an extensive higher 
education system and different approaches in place for external quality assurance may 
opt for a division of responsibilities. Such divisions may be regional, or based on the 
type of institution, or procedural, etc. Many different approaches are possible. The ESG 
do not address the division of responsibility. 

At a first glance the responses to the question about responsibility (table 8) indicate 
a largely homogeneous landscape in so far as two-thirds of the agencies are solely 
responsible for external quality assurance. On closer inspection the situation becomes 
even more homogeneous. Considering the 20 responding agencies which claim not to 
be the only responsible actor in the field of external quality assurance, by discounting 
the three internationally established agencies, and by observing that 10 out of the 
remaining 17 represent only two national systems (namely the federally structured 
systems of Spain and Germany), there are only seven national systems which do not 
assign responsibilities in quality assurance to only one agency. 

Table 8. 


IS YOUR AGENCY THE ONLY COMPETENT/RESPONSIBLE BODY FOR THE 
EXTERNAL QUALITY ASSURANCE OF HIGHER EDUCATION INSTITUTIONS 
BASED IN ITS PRIMARY DOMAIN? 

RESPONSE 

COUNT 

RESPONSE % 

Yes 

28 

58.3 

No 

20 

41.7 

Comment 

20 


answered question 

48 

skipped question 

3 


This does not mean, however, that in those systems more than one agency bear the 
same responsibilities. On the contrary, the only national quality assurance systems 
with a somehow competitive environment, with agencies having exactly the same 
responsibilities, are the German and the Dutch ones, and to some extent also the 
Spanish. In the other systems with more than one agency, the agencies have different 
responsibilities and thus do not compete with one another. 

2.3.2 QUALITY ASSURANCE AS CORE FUNCTION OF THE AGENCIES 

ESG standard 3.3 states that agencies should undertake external quality assurance 
activities on a regular basis. 

This standard addresses the expected level of experience (and professionalism) in 
undertaking external quality procedures. Given the crucial role of quality assurance 
stressed by ministers in Berlin in 2003, it is natural that they, as well as higher 
education institutions and other stakeholders, would demand a high level of quality 
and professionalism of external quality assurance agencies. An agency needs to 
perform external quality assurance as a core function and on a regular basis to reach an 
adequate level of professionalism and reliability in terms of policy, procedures, skills of 
staff and experts, etc. Moreover, compliance with ESG standard 3.3 is necessary for an 
agency to be able to meet the other standards of Part 3 of the ESG. 

Given the fact that almost all national quality assurance systems in the countries 
covered by the responding agencies are established by legislation with agencies having 


defined responsibilities, it is not surprising that respondent agencies normally conduct 
external quality assurance regularly, as their main task. Only five out of 47 agencies 
(including one network of agencies) responded that they do not perform external 
quality assurance regularly, mostly due to the fact that they have only recently been 
established or have recently changed their responsibility. 

2.4 Activities 

2.4.1 TYPE OF EXTERNAL QUALITY ASSURANCE 

ESG standard 3.3 does not specify the type of external quality assurance that should be 
applied by the agencies. It only states that external quality assurance shall take place 
“at institutional or programme level”. The standard is in line with one of the main 
principles of the ESG - the principle of tolerance of diversity: the ESG are “designed to 
be applicable to all higher education institutions and quality assurance agencies in Europe, 
irrespective of their structure, function and size, and the national system in which they 
are located (ESG, p.i2)[. . .] The standards [. . .] reflect basic good practice across Europe 
in external quality assurance, but do not attempt to provide detailed guidance about what 
should be examined or how quality assurance activities should be conducted.” (ESG, p.15) 

The principle of tolerance of diversity is essential not least because of the 
development of external quality assurance since the early nineties. Although the main 
principles developed in the EU pilot project “Quality Assessment in the Field of Higher 
Education”, especially the four-stage-model, still form a common basis for external 
quality assurance, the currently applied procedures vary in detail as regards the design 
of the procedures. 

Amongst the respondents, the most frequent (dominant) types of procedures are 
‘evaluation’ and ‘accreditation’. ‘Audit’ and other types of procedure follow far behind 
(table 9). 


Table 9. Regularly carried out external quality procedures 


HOW FREQUENTLY ARE THE FOLLOWING TYPES OF EXTERNAL QUALITY 

REGULARLY 

RESPONSE 

PROCEDURES CARRIED OUT BY YOUR AGENCY? 


COUNT 

Evaluation of subjects 

5 

33 

Evaluation of programmes 

23 

35 

Evaluation of institutions 

14 

35 

Evaluation of themes 

2 

31 

Audit at subject level 

1 

29 

Audit at programme level 

7 

32 

Audit at institutional level 

13 

32 

Audit at thematic level 

1 

28 

Accreditation of subjects 

3 

30 

Accreditation of programmes 

25 

38 

Accreditation of institutions 

13 

34 

Accreditation of themes 

2 

29 

Benchmarking of subjects 

2 

28 

Benchmarking of programmes 

3 

30 

Benchmarking of institutions 

1 

30 

Benchmarking of themes 

0 

27 

Other types of combined approaches (please specify) 

5 

25 


The main findings to be drawn from the responses are: 

• The most common external quality procedures are evaluation and accreditation 
followed by audit, others such as benchmarking and ranking are rarely part of the 
agencies’ remits. 

• About two-thirds of the agencies use programme-level procedures in their 
external quality assurance whereas institutional-level procedures are only applied 
by about 40% of the agencies. These ratios do not differ between evaluation and 
accreditation. Audit is understandably more uncommon at the programme level. 

• At a closer look, in almost all cases the specific external quality procedures are 
carried out either regularly or never. The only significant exceptions to this are 
evaluation of institutions and of themes which are carried out occasionally by 
approximately 20% of respondents. 

These major findings give a picture of well developed systems with equally well 
established regulations concerning the choice of approach. 

On closer inspection the agencies’ responses contain more interesting information. 
Most of the respondent agencies (about 90%) are not confined to only one type of 
external quality procedure. Moreover, some of them, for example HAC, even conduct 
most types of procedures (evaluation, audit and accreditation). Considering the most 
frequently applied approaches, reveals the following correlations: 

• Agencies that conduct programme evaluation also conduct - at a rate of fifty 
percent - accreditation of programmes and evaluation of institutions. 

• Agencies that conduct accreditation of programmes also conduct - at a rate of fifty 
percent - evaluation of programmes and accreditation of institutions. 

• Almost every agency that accredits institutions also accredits programmes, and 
half of them also evaluate programmes and institutions, and audit institutions. 


• Almost two-thirds of the agencies conducting audits at the institutional level also 
evaluate programmes, and half of them also accredit programmes, and evaluate 
and accredit institutions. 

In all these cases the respective procedures are applied regularly. 

Hence it is quite common for agencies to apply different types of external quality 
procedures. A possible explanation for this pattern may be the fact that quality 
assurance in higher education has undergone a highly dynamic evolution and that this 
process is continuing. Furthermore, it may be conjectured that the optimal approach 
to external quality assurance and the configuration of the national quality assurance 
system is situational: i.e. the approach is determined at least partly by the national 
context. 

This is not surprising considering that quality is a relative rather than an absolutely 
definable concept. Agencies, governments and stakeholders may have somewhat 
different views on quality and particularly on the most effective ways to enhance 
and assure it. The tolerance for diversity in the approach to quality assurance and 
especially for the assignment of more than one responsibility to one agency may be 
influenced by the national context and perception of external quality assurance. This 
seems significant, for instance, in the cases where accreditation of programmes has 
been added to existing types of quality assurance to meet the requests of students and 
employers or even tax payers. 

These findings are even more interesting when combined with the analysis of the 
consequences of the different types of procedures. 

2.4.2 FORMAL CONSEQUENCES OF EXTERNAL QUALITY ASSURANCE 

It is no surprise that accreditation procedures at programme level as well as at 
institutional level have significant consequences for formal approval of programmes 
and institutions respectively as well as for funding (table 10). There are formal 
consequences for approval and/or funding in 90% of cases. However, given the 
traditional definition of accreditation it is rather surprising that every tenth agency 
responded that there are no formal consequences. It is equally interesting to find 
out that evaluation at programme level and at institutional level as well as audit at 
institutional level also lead to formal consequences in between 54 and even 70% of the 
cases. 

This high level of obligation (flowing from consequences) corresponds to the fact 
that higher education institutions are formally required to address recommendations at 
a rate of approximately 75% in evaluation procedures and audits, and at approximately 
87% in accreditation procedures. 

Recalling the discussions about pros and cons of the different approaches to external 
quality assurance and especially the theoretical concepts that form the basis for the 
different approaches, it is important to recognise that the consequences are, in most 
cases, defined independently of the type of external quality assurance. The main 
objectives of external quality assurance are to approve programmes and/or institutions 
and to inform decisions on funding, and all the most common types of external quality 
procedures are applied for these purposes. It is also worth noting that the term “formal 
consequences” does not in every case mean “direct” or automatic consequences but 
rather preparation for a formal decision made by another party. 


Table 10. Consequences of external quality procedures 



FUNDING 

FORMAL 

STATUS/ 

APPROVAL 

FUNDING 
+ FORMAL 
STATUS/ 
APPROVAL 

OTHER 

NO FORMAL 
CONSEQUENCES 

Programme Evaluation 

0 

11 

5 

3 

6 

Institutional Evaluation 

0 

12 

4 

3 

4 

Programme Accreditation 

0 

18 

7 

1 

2 

Institutional Accreditation 

0 

14 

3 

0 

2 

Programme Audit 

0 

5 

0 

2 

2 

Institutional Audit 

1 

7 

1 

3 

3 


2.4.3 CYCLE OF EXTERNAL QUALITY ASSURANCE PROCEDURE 

The principle that external quality assurance is periodic as stipulated in ESG standard 
2.7 is not controversial. External quality assurance especially contributes to quality 
enhancement when it is done on a cyclical basis and not “once in a lifetime”. 

The responses to the question about the length of the cycle draw a very clear picture: 
most external quality procedures are carried out in cycles of five or six years. 

Figure 11. Activity cycle in years 
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2.4.4 CHANGE OF EXTERNAL QUALITY ASSURANCE APPROACH 

As mentioned before, one of the most striking features of external quality assurance in 
European higher education is the dynamism of its development over the last ten years. 
In the light of this, it not surprising that three-quarters of the agencies responded that 
they have changed their quality assurance approach recently or that they are about 
to do so in the near future (table 12). Nine out of the 36 agencies which answered 
“yes” only made some small adjustments to their procedures. The other 27 agencies 
that made significant changes can be divided into three groups. First, and this is not 
surprising, eight agencies replied that they modified their approach in order to align it 
with the ESG. The second group, consisting of three agencies, changed their approach 


mostly by switching to another type of procedure. Interestingly enough the last group of 
the remaining 16 agencies added a new type of procedure to the existing one(s), instead 
of replacing the “old” one, which corresponds to the above mentioned finding that 
many agencies apply more than one external quality assurance approach. 


Table 12. 


HAS YOUR AGENCY SIGNIFICANTLY CHANGED ITS APPROACH TO 
EVALUATION IN THE PAST FIVE YEARS OR IS IT PLANNING TO CHANGE ITS 
APPROACH? 

RESPONSE 

COUNT 

RESPONSE % 

Yes 

36 

75 

No 

12 

25 

answered question 

48 

skipped question 

3 


2.5 Resources 

2.5.1 HUMAN AND FINANCIAL RESOURCES 

The ESG assert in standard 3.4 that agencies should have adequate and proportional 
resources, both human and financial, to enable them to organise and run their external 
quality assurance processes) in an effective and efficient manner, with appropriate 
provision for the development of their processes and procedures. 

The meaning of this standard is twofold: on the one hand, agencies should have 
appropriate resources to effectively and efficiently conduct their external quality 
procedures. On the other hand, the provision of resources should enable agencies 
to further develop their processes and procedures. This second facet is linked with 
the standard on accountability procedures, because the (further) development of 
quality assurance procedures should be based both on internal and external feedback 
and reflection mechanisms in order to inform and support the development and 
improvement actions. Both aspects were covered by the survey. The results (in 
percentage) are shown in figure 13. 

Adequate and proportional 
financial resources for the 
continual evaluation and 
development of processes and 
procedures 

■ Adequate and proportional 
human resources for the 
continual evaluation and 
development of processes and 
procedures 

■ Adequate and proportional 
financial resources to organise 
and run external quality 
procedures in an effective and 
efficient manner 

■ Adequate and proportional 
human resources to organise 
and run external quality 
procedures in an effective and 
efficient manner 


Figure 13. Adequate resources 
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The average rating provides, for each of the question’s propositions, a numerical 
expression of the respondent’s overall level of agreement with it. The highest average 
rating (3.78) concerns provision with adequate and proportional financial resources to 
enable the agencies to organise their external quality procedures in an effective and 
efficient manner. The lowest average rating (3.43) relates to the provision with adequate 
and proportional human resources for the continual evaluation and development of the 
agencies’ processes and procedures. The average rating pertaining to provision with 
adequate and proportional human resources to enable the agencies to organise their 
external quality procedures in an effective and efficient manner is 3.67. The average 
rating applying to provision with adequate and proportional financial resources for the 
continual evaluation and development of their processes and procedures is 3.52. 

On the whole, the results of the survey show that there is agreement that the 
provision of human and financial resources is adequate but it falls short of being strong 
agreement. In analysing the data one has to bear in mind the developmental stage of 
agencies. Three agencies, for example, mentioned that they have just recently been 
established and that in the years to come they will need to increase their human and 
financial resources. One agency commented that it is severely underfinanced and 
understaffed and that this is due to the fact that the government budget has for several 
years allotted 10-20% less funds to it than requested. 

The ESG and the corresponding external review of agencies aim inter alia to ensure 
that agencies act professionally and with a high level of expertise and credibility across 
Europe and to encourage external quality assurance capacity building. The continuous 
improvement and the establishment of internal quality management systems is not 
only an important challenge for higher education institutions but also for external 
quality assurance agencies. In this context, research and development activities play 
a crucial role within agencies. Therefore, it is quite striking that the lowest rating 
average pertains to provision with adequate and proportional human resources for the 
continual evaluation and development of the external quality assurance processes and 
procedures. 

External quality assurance agencies operate in a socially and technically (referring to 
QA methods) complex domain where there can be tensions between generally accepted 
principles. With governments, higher education institutions and quality assurance 
agencies all on the playing field, there can be uncertainty about who is in charge of 
what? Strains can arise in relation to the following: 

• the autonomy of higher education institutions; 

• the ‘two-pillar concept’ involving higher education institutions’ internal quality 
management and external quality assurance; 

• the division of responsibility; 

• governmental steering mechanisms; 

• diverse interests of students; 

• societal, economic, methodological, organisational and content aspects of external 
quality assurance. 

In this complex and sensitive environment external quality assurance agencies should 
be enabled - in order to gain credibility and to ensure the quality of their own activities 
- to self-critically reflect on their own activities based on both internal and external 
inputs. This requires the allocation of appropriate human resources. 


2.5.2 COMPOSITION OF STAFF 

The question put to the agencies was: How many full-time equivalent staff members 
are (directly) employed by (or seconded to) your Agency? The results of the survey are 
shown in table 14. 

Table 14. Composition of staff 


HOW MANY FULL-TIME EQUIVALENT STAFF MEMBERS 
ARE (DIRECTLY) EMPLOYED BY (OR SECONDED TO) 
YOUR AGENCY? 

RESPONSE 

AVERAGE 

RESPONSE 

TOTAL 

RESPONSE 

COUNT 

Number of external quality procedures officers 

4.9 

214 

44 

Number of staff covering functions that are unrelated 
to external quality procedures 

8.9 

382 

43 

Number of administrative staff 

24 

891 

37 

Number of management staff 

3 

76 

23 

Number of research/development officers 

1.2 

28 

23 

Number of trainees/interns 

152 

2737 

18 

Other (please specify in answer to next question) 

19.8 

415 

21 

answered question 

45 

skipped question 

6 


When interpreting the table it is useful to consider the following detail. The number of 
management staff varies from o (one agency) to 29; 38 agencies reported the number 
of management staff to be in the range of one to ten; six agencies indicated that the 
number of management staff is in the range of 13 to 29. In the three cases where it 
was indicated that the number of management staff is in the range of 25 to 29, it is 
reasonable to assume that the members of the agencies’ decision making bodies were 
included in number of the management staff of agencies. 

Thirty five agencies answered that the number of administrative staff is in the range 
of one to ten. The number of administrative staff of three agencies is in the range of 11 
to 13; five agencies have between 24 and 70 administrative staff members. 

Three agencies indicated that they do not employ (or second) any external quality 
procedures officers; 27 agencies mentioned that the number of external quality 
procedures officers is in the range of one to twenty and the number of external quality 
procedures officers of five agencies is in the range of 20 to 48; two agencies answered 
that the number of external quality procedures officers is 224 and 300 respectively. In 
the latter two cases it is reasonable to assume that the members of the expert panels 
were added to the number of the external quality procedures officers of the agency. 

It is quite striking to note that only 13 agencies indicated that research and 
development officers are part of their staff. Within this group the number of such 
staff varies from one to 26. Eleven agencies answered that the number of research and 
development officers is in the range of one to seven and two agencies mentioned that 
the number is 17 and 26 respectively. 

Eighteen agencies have staff in category “other” the numbers vary between o 
and 1,400. This category covers staff members like internal quality assurance staff, 
international officers, officer for legal affairs, IT-experts, students, members of the 
Boards/Councils, public relation officers, scientific collaborators for external procedures 
and development and research, officers for Communication and members of the expert 


panels. It is noteworthy that agencies make use of quite high numbers of members for 
expert panels, for example ZEvA (1,400) and NEAA (1,300). 

All in all, apart from the inconsistent interpretation of the meaning of the different 
categories of staff, the data show that external quality assurance agencies across Europe 
differ quite a lot in terms of structure, function and size. It is reasonable to assume 
that this heterogeneous picture is also reflected in the size of the agencies’ primary 
domain and in the remit or scope of its activities. The relatively low proportion of 
agencies with research and development officers as staff members could be an area 
for development in light of the importance of implementing and maintaining internal 
quality management systems. 

2.5.3 FUNDING OF AGENCIES 

As mentioned above, almost every respondent agency is recognised explicitly in 
legislation in the whole of its primary domain and/or is recognised by the competent 
authorities established for that purpose. This means that, in almost all the countries 
which are covered by the responding agencies, the initiative to set up external quality 
procedures has either been taken, or supported by government. Governments evidently 
perceive external quality assurance to be an effective instrument for assuring and 
enhancement of quality of higher education. The high priority status that governments 
grant to quality assurance is also reflected in the funding of the agencies. 

The questionnaire asked the agencies to indicate the approximate percentage of 
their funding received from a selection of sources (the sum of the entries was 100). The 
findings are presented in table 15. 

Table 15. Funding of agencies 


FUNDING SOURCES 

RESPONSE 

AVERAGE 

RESPONSE TOTAL 

RESPONSE 

COUNT 

Government 

70.63 

2896 

41 

HEIs (fixed subscriptions only) 

37.53 

563 

15 

Fees (for specific activities) 

32.25 

903 

28 

Grants (other than government) 

2.25 

27 

12 

Other (please specify in answer to next question) 

11.72 

211 

18 

answered question 

46 

skipped question 

5 


Government funding is the main source of funding for agencies, followed by higher 
education institutions. Grants are the third most important source of funding. Perusal 
of the responses revealed the details that are shown in figures 16-19. 


Figure 16. Governmental funding 

Funding by the government between i 

2% and 8%, 1 agency 

Funding by the government between i 

10% and 34%, 2 agencies 

Funding by the government between i 

52% and 98%, 15 agencies 



Figure 17. HEI funding 

No funds (0%) from HEI, 5 agencies 


Completely funded (100%) by the 
HEI, 2 agencies 


Funding by the government ' 

between 2% and 50%, 5 agencies 



Funding by the government between 
77% and 90%, 3 agencies 


Figure 18. Fee funding 

No fee (0%) received, . 

5 agencies 

Funding by fees between 1 — 
2% and 8%, 6 agencies 

Funding by fees between > — 
10% and 34%, 9 agencies 



Completely funded (100%) by fees, 
2 agencies 


Funding by fees between 52% 
and 98%, 6 agencies 


Figure 19. Grant funding 


No grants (0%) received, 
8 agencies 



Funding by grant between 5% 
and 10%, 4 agencies 


Eighteen of the responding agencies stated that a specific part of their funding comes 
from sources other than the proposed categories. The percentage for funding from 
other sources varies from 1% to 75% and covers for example the following sources: 
international projects, EU Commission’s Socrates project grants, European structural 
funding, professional organisations, membership fees, state commissioned projects and 
contracts to develop QA in other countries. 

2.6 Mission statement 

The ESG require in standard 3.5 that agencies should have clear and explicit goals 
and objectives for their work, contained in a publicly available statement. The 
corresponding guidelines suggest that these statements should describe the goals and 
objectives of agencies’ quality assurance processes; the division of labour between the 
agency and relevant stakeholders especially the higher education institutions; and the 
cultural and historical context of their work. The statements should make clear that 
the external quality assurance process is a major activity of the agency and that there 
exists a systematic approach to achieving its goals and objectives. There should also be 


documentation to demonstrate how the statements are translated into a clear policy 
and management plan. 

Almost all responding agencies publish a statement of their goals and objectives 
and a statement setting out a systematic approach to external quality assurance (see 
table 20). 63.8% of the responding agencies publish a statement explaining the division 
of responsibility for quality procedures between the agency and higher education 
institutions and 59.6% of agencies publish a general policy and management plan. 

Twenty agencies indicated that they publish additional kinds of information to those 
proposed by the questionnaire. This information covers, for example, the following: 
composition of the quality assessment committees; report of activities; assessment 
guides; self-evaluation and external evaluation reports; quality policy; labour market 
reports; global annual report on quality assurance on higher education; code of 
ethics; procedures; standards and guidelines for accreditation and evaluation; quality 
management system; annual reports; manual for training external review teams; etc. 

Table 20. Mission statement 


WHAT KIND OF GENERAL INFORMATION ABOUT ITSELF DOES YOUR 
AGENCY PUBLISH FOR THE BENEFIT OF STAKEHOLDERS? 

RESPONSE 

COUNT 

RESPONSE % 

Statement of the goals and objectives of the Agency 

42 

89.4 

Statement setting out the systematic approach to external quality 
assurance 

40 

85.1 

Statement explaining the division of responsibility for quality 
procedures between the Agency and higher education institutions 

30 

63.8 

Statement of general policy and management plan 

28 

59.6 

Other (please specify) 

20 

42.6 

answered question 

47 

skipped question 

4 


2.7 Independence 

The ESG determine in standard 3.6 that agencies should be independent to the extent 
that they have both autonomous responsibility for their operations and that the 
conclusions and recommendations made in their reports cannot be influenced by third 
parties, such as higher education institutions, ministries or other stakeholders. The 
corresponding guidelines outline some specific measures, viz.: 

• The agency’s operational independence from higher education institutions 
and governments is guaranteed in official documentation (e.g. instruments of 
governance or legislative acts); 

• The definition and operation of its procedures and methods, the nomination and 
appointment of external experts and the determination of the outcomes of its 
quality assurance processes are undertaken autonomously and independently from 
governments, higher education institutions, and organs of political influence; 

• While relevant stakeholders in higher education, particularly students/learners, 
are consulted in the course of quality assurance processes, the final outcomes of 
the quality assurance processes remain the responsibility of the agency. 

The issue of independence is regarded as being critically important and deserving of 
special attention, since it, among others, forms the basis for professionalism and thus 
the basis for trust. In September 2006 the ENQA Board held a seminar on membership 


issues and concluded that independence for the purpose of the ESG should be 
interpreted as “autonomous responsibility for operations, where conclusions in reports 
cannot be influenced by HEIs, ministries or other stakeholders”. 

In 2005-06 the Nordic Quality Assurance Network in Higher Education (NOQA, 
inch QA bodies from Norway, Sweden, Denmark, Finland and Iceland 10 ) carried out 
a project entitled “ESG in a Nordic perspective”. The aims of the project were: (i) to 
interpret and clarify the ESG for quality assurance agencies; (ii) to share and compare 
practices among the Nordic agencies; (iii) to reflect on how organisations, processes 
and procedures could be enhanced having regard to the new European standards; and 
(iv) to determine how the ESG for external QA agencies could be interpreted and made 
operational for assessment. 

Considering that the ESG are designed to be applicable to all quality assurance 
agencies in Europe, irrespective of their structure, function and size, and the national 
system in which they operate, the results of the NOQA project 11 raised many important 
and interesting questions about this ambitious level of generality of applicability viz. 
What role do national traditions and legislation play in respect of agencies’ compliance 
with the ESG? Do legislation and/or traditional approaches need to be changed? Can 
they be viewed as a reason for allowing exemption from elements of the ESG? What 
is the significance of the moderation of formal arrangements by informal practice? 

For instance, an agency can be formally independent from ministries and other 
stakeholders, but not independent in practice if the government places a high level of 
pressure on the agency through informal channels. Conversely an agency that operates 
independently in practice may not be formally independent. The findings of this survey 
can be related to the Quality Convergence Study, conducted by ENQA in 2005, which 
clearly showed, that although national quality assurance systems converge to some 
extent, the practicalities in national systems may differ without compromising shared 
principles. 12 

In the survey, two questions were put to the agencies addressing the issue of 
independence. The results of the survey are shown in tables 21 and 22. 

Table 21. Influence of stakeholders on reports 


IN YOUR OPINION, WHAT INFLUENCE, IF ANY, DO EXTERNAL STAKEHOLDERS 
HAVE ON THE CONCLUSIONS AND RECOMMENDATIONS MADE IN YOUR 
AGENCY'S REPORTS? 

RESPONSE 

COUNT 

RESPONSE % 

External stakeholders may influence neither the conclusions nor the 
recommendations in reports 

30 

66.7 

External stakeholders may sometimes influence the recommendations in 
reports 

11 

24.4 

External stakeholders may sometimes influence the conclusions in reports 

10 

22.2 

Comment 

15 


answered question 

45 

skipped question 

6 


www.noqa.net 

www.noqa.net 

http://www.enqa.eu/files/Quality%20Convergence%20Study.pdf 


Thirty of the responding agencies (66.7%) indicated that external stakeholders may 
influence neither the conclusions nor the recommendations in reports. The possibility 
of occasional external stakeholders’ influence both on recommendations and 
conclusions in reports was reported by 21 agencies (46.6%). 

In analysing and interpreting the data related to occasional external stakeholders’ 
influence on recommendations and conclusions in reports, one has to take into 
account the comments made by respondents. Four agencies pointed out that external 
stakeholders are involved in committees and expert panels and that they are 
interviewed during the site visit, which leads to the fact that their views are taken into 
account in the final report. Two agencies commented that HEIs are provided with the 
possibility to make a response statement to the draft report of the expert panel which 
may get included in the final report. One agency mentioned that the results of students’ 
surveys, which are discussed by the accreditation board, may sometimes influence 
the recommendations in the reports. Another agency took the view that stakeholder 
involvement is a key principle of its system and that this helps to avoid attempts of 
external influence. It was also stated that, in very exceptional cases, if a question is 
politically sensitive, external stakeholders may sometimes influence the conclusions 
of the reports. One agency commented that the social milieus of the evaluation, 
academic and ministry worlds are very similar and that the same people during their 
career may hold positions in one of the three fields of activities. This leads to mostly 
positive, but also in some cases negative, interactions. Another agency which is at an 
early developmental stage pointed out that this will have to be checked in practice. 

In conclusion this means that only one agency mentioned influence by external 
stakeholder which could compromise its decisions. 

Table 22. Autonomy of agencies 


IN YOUR OPINION, WHAT SHARE, IF ANY, DO EXTERNAL STAKEHOLDERS 
HAVE IN RESPONSIBILITY FOR YOUR AGENCY'S OPERATIONS? 

RESPONSE 

COUNT 

RESPONSE 

% 

No share, my Agency has autonomous responsibility 

37 

78.7 

Other 

10 

21.3 

answered question 

47 

skipped question 

4 


Thirty seven agencies (79%) indicated that external stakeholders do not have any 
share in responsibility for the agencies’ operations, i.e. that they have autonomous 
responsibility. Ten agencies (21%) pointed out that external stakeholders do have some 
share in responsibility for the agencies’ operations. However, external stakeholders’ 
share in responsibility for the agencies’ activities has to be interpreted having regard to 
the comments made by respondents under “other”. 

Four agencies mentioned that within their organisation structure, the membership of 
different bodies is drawn from stakeholder groups. For example (external) stakeholders 
are involved as members of committees and expert panels as well as of the decision 
making body. One of these agencies drew attention to the fact that, as members of the 
decision making body, the stakeholders act autonomously and do not represent any 
particular interests. Three agencies commented that stakeholder views are sought as 
part of consultation processes related to significant changes in policy and process, to 
improvement issues and to general working methods but cannot influence individual 


cases. Another agency pointed out that it has autonomous responsibility for external 
quality procedures but not for decisions. 

On the one hand, these specifications suggest that stakeholder involvement plays 
a crucial role in external quality assurance arrangements. On the other hand, they 
indicate that the agencies’ membership drawn from stakeholder groups should not be 
regarded as external stakeholders. In summary, stakeholder involvement should not be 
mistaken for influence of external stakeholders on the agencies operations. 


Chapter 3: Typical methodological 
approach to external quality 
assurance 

3.1 Introduction to Chapter 3. 

This chapter addresses typical methodological approach to external quality assurance 
and is divided into two parts. The first part (Section 3.2) describes the data and reports 
the detailed findings. The second part (Section 3.3) is a commentary on the main 
findings. 

If you are just interested in an overview of the main findings you can skip directly to 
Section 3.3. 

3.2 Description of the data 

3.2.1 THE DATA FOR QUESTIONS 25-38 

There were 47 respondents that had given answers to the set of questions from 25 to 38. 
This number was made up of 25 full members (members); seven Candidate members 
(Candidates); four Affiliates; 10 Associates and one other. In the analysis in some cases 
the responses are reported separately for two sub-groups: (i) Members and Candidates 
and (ii) Affiliates and Associates. 

However, not all of these questions were answered by all 47 respondents. In some 
cases this is because the question was not found relevant by all respondents. For 
example questions about programme-level procedures are only relevant to those 
agencies which use them. No further conclusions about other possible reasons for 
agencies skipping questions were drawn. Accordingly, when percentages are reported 
they relate to the number of respondents which addressed the question and not to the 
total number of respondents to the survey (51). 

In a number of questions respondents were invited to use a five-point rating scale 
(from one to five) when responding to items. Alternatively, they could indicate that a 
particular item is not relevant (i.e. that it is not an objective/function/activity of the 
agency). In such cases the latter indication is combined (valued at o) with the rating to 
produce an impact valued between o and five. 

The ratings of one respondent (or group of respondents) are not calibrated to those of 
another, thus there are no control questions. This means that a small difference in pairs 
of ratings or of average ratings is not necessarily significant. 

Some of the questions ask respondents to rate the “importance” of certain criteria. 
The meaning of “importance” is a little vague. One interpretation is ‘absolute 
importance’ but another might be relevance i.e. an item is regarded as unimportant if 
an agency has a very small share in the responsibility for evaluating it. 

Where data are tabulated they are normally sorted in order of percentage or average 
impact. Inspection of the tables will allow readers to tell at a glance which are the 
most/least significant items. 

Qualitative responses (where not anonymous) are attributed to an agency or a 
country (if unambiguous). In quite a few cases the “other” option was used by some 
respondents not to describe an “other” category but rather to make a comment. 


3.2.2 ON THE CONSTITUTION OF THE AGENCY’S BOARD OR COUNCIL 

The vast majority (94%) of agencies have a board or council (table 23). Two of those 
that do not have a Board/Council indicated that they had “other” arrangements in place 
viz. a “permanent working commission” (Czech Republic); and an “independent body 
whose members are approved by the Ministry of Education” (Russia). 

Table 23. 


DOES YOUR AGENCY HAVE A BOARD OR COUNCIL (I.E. A FORMALLY 
ESTABLISHED GOVERNING COMMITTEE)? 

RESPONSE 

COUNT 

RESPONSE % 

Yes 

44 

93.6 

No 

3 

6.4 

Other (please specify) 

6 


Total 

47 


skipped question 

4 


The most commonly represented stakeholder groups represented on respondents’ 
Boards/Councils (table 24) are “higher Education Institutions”; followed, in joint second 
place, by “industry and labour market representatives (union, employers, etc.)” and 
“students”. Those three stakeholder groups remain the most likely to be represented 
for the sub-groups of respondents with three or more groups represented and for those 
with five or more. 


Table 24. 


WHICH OF THE FOLLOWING STAKEHOLDER 
GROUPS IS REPRESENTED ON THE BOARD/ 
COUNCIL OF YOUR AGENCY? 

RESPONSES 
WITH THREE 
OR MORE 
GROUPS (%) 

RESPONSES 
WITH FIVE OR 
MORE GROUPS 
(%) 

RESPONSE 

COUNT 

RESPONSE 

% 

Higher education institutions 

91 

100 

37 

82 

Industry and labour market representatives 
(union, employers, etc.) 

63 

77 

22 

49 

Students 

69 

100 

22 

49 

Government (central/regional) 

50 

62 

17 

38 

International quality assurance community 

34 

54 

14 

31 

Other (please specify) 

28 

23 

14 

31 

Professional organisations 

41 

54 

13 

29 

National association of HEIs (such as rectors’ 
conference) 

41 

46 

13 

29 

National quality assurance community 

28 

46 

10 

22 

International association of HEIs (such as EUA) 

3 

0 

1 

2 

answered question 

45 

skipped question 

6 


Fourteen respondents selected the “other” category (again sometimes to comment 
rather than provide an alternative to those listed): in three cases this meant (in broad 
terms) academics; and in three cases it meant the research community (AERES, NEAA 
Bulgaria, ANECA.) 

Three more respondents who selected “other” distinguished between a person 
representing a group and a person affiliated to a group but not representing it e.g. 


“stakeholder groups are not represented on the Board: all Board members are individuals 
and only represent their personal views. The composition of the Board does however take 
into account the affinity of the different persons and seeks a balance between affinities with 
stakeholder groups. ”(NVAO) 

Figure 25 illustrates the distribution of the number of stakeholder groups (including 
“other”) represented on the agency’s board/council. It peaks at four stakeholder groups 
and the average number is 3.5. 

Figure 25. Number of stakeholders 
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3.2.3 ON THE DIFFERENT FUNCTIONS OF THE AGENCY AND THEIR 
RELATIVE IMPORTANCE 

Table 26 lists the functions of agencies in order of average impact and also displays the 
average rating. The difference between impact and rating is that the former includes 
“not a function” responses with an impact of zero whereas the latter does not. Figure 27 
provides a graphical illustration of the data. 

The functions that have the highest impact are: “quality assurance”; “quality 
enhancement/improvement”; “external quality assurance of programmes”; “external 
quality assurance of institutions”; “collecting/ disseminating information on quality of 
HE” and most respondent agencies have all of those as functions. A total of 34 out of 44 
respondents (77%) include all five of these as functions. 

The function with the lowest average impact is “deciding on the funding of higher 
education”. 48% of respondents do it but even among those its relative average 
importance (moderately important to important) is least among the alternatives 
provided. 

Exactly 50% of respondents have “recognition of national diplomas” as a function 
and among those it is considered important to very important. 

The function “recognition and licensing of higher education institutions” is 
implemented by 68% and on average it is considered important to very important by 
those. 

“Development and maintenance of discipline standards” is a function of 68% and on 
average it is considered moderately important to important by those. 

A majority of respondent agencies develop and maintain qualification frameworks 
(63%). 


The average ratings given to the functions (where a function of the respondent’s 
agency) by Members and Candidates are comparable (within about 0.5 standard 
deviations) to those given by Affiliates and Associates. The greatest difference 
(measured in standard deviations) occurs for “development and maintenance of 
discipline standards”; and “collecting/disseminating information on quality of HE...” 
both of which receive a higher average rating from Affiliate/Associate respondents. 

Considering average impact of functions reveals greater differences between the two 
respondent groups. The most significant impact differences between the two groups 
are for “development and maintenance of discipline standards”; and “deciding on the 
funding of higher education”, where the average impact is higher for Affiliate/Associate 
respondents than for Member/Candidate respondents (table 26). 

Table 26. 


INDICATE THE RELATIVE IMPORTANCE TO YOUR 
AGENCY'S MISSION OF THE FOLLOWING (WHERE 
THEY ARE FUNCTIONS OF YOUR AGENCY). 

UNIMPORTANT 

OF LITTLE 
IMPORTANCE 

MODERATELY 

IMPORTANT 

IMPORTANT 

VERY IMPORTANT 

NOT A FUNCTION 
OF MY AGENCY 

RATING AVERAGE 

IMPACT AVERAGE 

RESPONSE 

COUNT 

Quality assurance 

0 

0 

2 

7 

34 

0 



43 

Quality enhancement/improvement 

0 

0 

2 

6 

34 

1 



43 

External quality assurance of programmes 

0 

2 

3 

5 

32 

2 



44 

External quality assurance of institutions 

0 

0 

3 

8 

29 

4 



44 

Collecting/disseminating information on quality of 
HE... 

0 

3 

7 

15 

17 

2 



44 

Recognition and licensing of higher education 
institutions 

0 

0 

4 

7 

17 

14 



42 

Development and maintenance of the qualification 
framework 

0 

0 

9 

6 

12 

16 



43 

Development and maintenance of discipline 
standards 

2 

1 

8 

10 

8 

14 



43 

Recognition of national diplomas 

0 

1 

1 

4 

14 

22 



42 

Deciding on the funding of higher education 

3 

2 

5 

5 

6 

21 



42 

Other 









5 

answered question 

44 

skipped question 

7 


Figure 27. Importance of your Agency's mission of the following where they are functions of 
your agency 


Deciding on the funding of HE 
Recognition of national diplomas 

I Development and maintenance of 

dicipline standards 

Development and maintenance of the 

I *"~ = ===== ^- - qualification framework 

Recognition and licencing of HEI 

I Collecting/disseminating information 

I on quality of HE . . . 

External QA of institutions 
External QA of programmes 
Quality enhancement/improvement 
Quality assurance 

0 % 20 % 40 % 60 % 80 % 100 % 

Unimportant ■ Important 

Of little importance ■ Very important 

■ Moderately important ■ Not a function of my Agency 


3.2.4 ON THE DIFFERENT OBJECTIVES OF THE AGENCY’S EXTERNAL 
QUALITY PROCEDURES AND THEIR RELATIVE IMPORTANCE TO ITS 
MISSION 

“Transparency” and “quality improvement/enhancement” are the highest rated 
objectives of the agencies’ quality procedures. All respondents rate them as either 
important or very important. “Accountability” and “accreditation” come next. 
Nevertheless, three agencies (NVAO, FIBAA, and Swedish NAHE) do not have 
“accountability” as an objective of their quality procedures. 

63% consider “student (or prospective student) protection” to be important or very 
important. 

14% consider the estimation of graduate learning outcomes not to be an objective of 
their procedures. Most of those who have it as an objective, consider it to be important 
to very important. 


Table 28. 


INDICATE THE RELATIVE IMPORTANCE 
TO YOUR AGENCY'S MISSION OF THE 
FOLLOWING WHERE THEY ARE OBJECTIVES 
OF THE MAIN TYPE OF EXTERNAL QUALITY 
PROCEDURES UNDERTAKEN BY YOUR 
AGENCY. 

UNIMPORTANT 

OF LITTLE 
IMPORTANCE 

MODERATELY 

IMPORTANT 

IMPORTANT 

VERY IMPORTANT 

NOT AN 
OBJECTIVE 

RATING AVERAGE 

IMPACT AVERAGE 

RESPONSE 

COUNT 

Transparency 

0 

0 

0 

7 

37 

0 

4.84 

4.84 

44 

Quality improvement/enhancement 

0 

0 

0 

11 

33 

0 

4.75 

4.75 

44 

Accountability 

0 

0 

3 

14 

24 

3 

4.51 

4.20 

44 

Accreditation 

0 

1 

2 

4 

32 

5 

4.72 

4.18 

44 

Student (or prospective student) protection 

0 

1 

3 

10 

22 

8 

4.47 

3.66 

44 

Estimation of graduate learning outcomes 

0 

2 

4 

15 

17 

6 

4.24 

3.66 

44 

Make international comparisons with similar 
institutions/ programmes 

1 

2 

8 

16 

8 

8 

3.80 

3.09 

43 

Make national comparisons with similar 
institutions/programmes 

3 

1 

11 

13 

7 

9 

3.57 

2.84 

44 

Ranking 

5 

7 

4 

0 

1 

24 

2.12 

0.88 

41 

Other 









7 

answered question 

44 

skipped question 

7 


The average ratings given to the objectives (where an objective) by the Member/ 
Candidate respondents is comparable (within about 0.5 standard deviations) to that 
given by Affiliate/Associate respondents. The greatest difference (measured in standard 
deviations) occurs for “estimation of graduate learning outcomes” which receives a 
lower average rating from Affiliate/Associate respondents. 

Considering average impact reveals greater differences between the two groups. The 
most significant differences are for “ranking”; and “student (or prospective student) 
protection” where the average impact is higher for Affiliate/Associate respondents. 

One particularly noteworthy “other” objective is “complying with the Higher 
Education Regulation Review Group (HERRG) Concordat to ensure regulation is 
appropriate to the level of assurance sought” (QAA, UK). 

3.2.5 ON THE EXPERT PANELS AND THE ESTABLISHMENT AND 
DISSEMINATION OF THE AGENCY’S EXTERNAL QUALITY PROCEDURES 
AND CRITERIA INCLUDING THE INVOLVEMENT OF STAKEHOLDERS 

Almost all quality assurance agencies publish relevant processes, criteria and 
procedures prior to execution of external quality procedures (93%). Three respondent 
agencies [two ENQA members (AERES and an anonymous one) and one Candidate 
(AQA)] do not (table 29). 


Table 29. 


DOES YOUR AGENCY ALWAYS PUBLISH RELEVANT PROCESSES, CRITERIA 
AND PROCEDURES PRIOR TO EXECUTIONS OF EXTERNAL QUALITY 
PROCEDURES? 

RESPONSE 

COUNT 

RESPONSE % 

Yes 

43 

93.5 

No 

3 

6.5 

answered question 

46 

skipped question 

5 


A range of stakeholders have a formal role in the specification of processes and criteria 
for external quality assurance (table 30). “Higher education institutions” top the list 
followed by “government”. Only one response indicates “higher education institutions” 
as the only stakeholder group with a formal role in this process. 

Table 30. 


WHICH OF THE FOLLOWING STAKEHOLDERS HAVE A FORMAL ROLE IN THE 
SPECIFICATION OF PROCESSES AND CRITERIA FOR THE EXTERNAL QUALITY 
ASSURANCE OF HIGHER EDUCATION INSTITUTIONS IN YOUR PRIMARY 
DOMAIN? 

RESPONSE 

COUNT 

RESPONSE % 


MEMBERS AND 
CANDIDATES 

AFFILIATES AND 
ASSOCIATES 

Higher education institutions 

63% 

57% 

29 

62 

Government (central/regional) 

59% 

43% 

26 

55 

Student representatives 

56% 

50% 

25 

53 

Quality assurance agencies 

53% 

43% 

24 

51 

National association of HEIs (such as 
rectors’ conference) 

44% 

43% 

20 

43 

Industry and labour market 
representatives (union, employers, 
etc.) 

44% 

21% 

17 

36 

Other (please specify) 

28% 

50% 

16 

34 

Professional organisations 

34% 

29% 

15 

32 

International association of HEIs 
(such as EUA) 

16% 

21% 

8 

17 

answered question 

47 

skipped question 

4 


The most pronounced difference between the Member/Candidate respondent group 
and the Associate/Affiliate group is in the percentage giving a formal role to “industry 
and labour market representatives (union, employers, etc.)” which is 44% and 21% 
respectively. 

In more than half of all respondent agencies “government”, “quality assurance 
agencies” and “student representatives” are involved in the specification of processes. 
It would not be uncommon also to involve “industry and labour market” (36%) and 
“professional organisations” (32%). In a limited number of cases (four out of 47, three 
of which are ENQA full members) “government” is the sole stakeholder group with a 
formal role in specification. 


Three respondent agencies, which did not select any of the stakeholder options or 
just the “agency” option, make the point that stakeholders, while not having a formal 
role in the specification, would be consulted by the agency (Austrian FHR, NVAO, 
and NEAA Bulgaria). Exploring the comments under “other”, reveals that at least 
one respondent who indicated stakeholder group involvement might be in a similar 
situation with stakeholders involved “only via their participation in the membership of 
the committee” (HAC Hungary). 

Other stakeholders with a formal role include: Federal Service of Supervision in 
Education and Science (Russia); and “International experts” (German Accreditation 
Council.) 

The final decision on the specification of processes and criteria for external quality 
procedures (EQPs) is likely to be made by the relevant agency (79%). Table 31 illustrates 
the pattern of responses. Note that respondents frequently chose more than one option 
(which can mean that different types of decisions have different decision makers but 
could also mean a joint decision). In 62% of cases it is the agency alone that makes the 
final decision. In two cases it is the government alone. 

When neither the agency nor government make the final decision there is an 
external authority in four cases (Serbia, Russia, Croatia and Switzerland). 

There is a significant difference between the two respondent groups in the 
percentage of respondents indicating that the agency makes the final decision. 

Table 31. 


WHICH BODY MAKES THE FINAL DECISIONS IN THE SPECIFICATION OF 
PROCESSES AND CRITERIA FOR THE EXTERNAL QUALITY ASSURANCE 
OF HIGHER EDUCATION INSTITUTIONS IN YOUR PRIMARY DOMAIN? 

RESPONSE 

COUNT 

RESPONSE % 


MEMBERAND 

CANDIDATE 

ASSOCIATE 
AND AFFILIATE 

The relevant quality assurance agency 

88% 

57% 

37 

79 

Other (please specify) 

13% 

57% 

12 

26 

Government (central/regional) 

19% 

21% 

10 

21 

answered question 

47 

skipped question 

4 


Another way in which stakeholders are involved in the agency’s operations is when they 
may nominate/suggest members of the expert panel (table 32). 

89% of respondents indicated that the quality assurance agency may nominate/ 
suggest experts and 41% of those do so exclusively. The next most frequent responses 
are: “higher education institutions”; “students”; and “industry and labour market 
representatives (union, employers, etc.)”. 

In four cases (Sweden, Latvia, Austrian FH Council, NVAO) “quality assurance 
agencies” with which the agency cooperates including those in other countries may 
nominate experts. One respondent mentioned the European Student’s Union (ESU). 

One respondent made the point that HEIs have the right to “comment on 
the make-up of panels” (ECCE, UK) but not to suggest/nominate. Another 
respondent indicated that “external evaluators are selected on the basis of [an] open 
[advertisement]” (Serbia). In Russia the Federal Service of Supervision in Education and 
Science may suggest potential members to the expert panels, while the agency (NAA) 
certifies experts and compiles the expert database. 


A higher percentage (62%) of Affiliate/Associate respondents receive nominations/ 
suggestions for expert panel members from HEIs than Member/Candidate respondents 
(47%) and this imbalance is reversed for students where the percentages are 34% and 
23% respectively. 

Table 32. 


WHO MAY SUGGEST/NOMINATE POTENTIAL MEMBERS OF THE EXTERNAL 
EXPERTS PANELS USED BY YOUR AGENCY? 

RESPONSE 

COUNT 

RESPONSE 0 /. 


MEMBER AND 
CANDIDATE 

ASSOCIATE AND 
AFFILIATE 

Your Agency 

91% 

85% 

41 

89 

Higher education institutions 

47% 

62% 

24 

52 

Students 

34% 

23% 

14 

30 

Industry and labour market 
representatives (union, employers, etc.) 

28% 

23% 

13 

28 

National association of HEIs (such as 
rectors’ conference) 

25% 

31% 

12 

26 

Other (please specify) 

25% 

31% 

12 

26 

The institution that is the subject of the 
external quality procedure involving the 
panel 

25% 

15% 

10 

22 

Professional organisations 

19% 

23% 

9 

20 

Government (central/regional) 

6% 

23% 

5 

11 

International association of HEIs (such 
as EUA) 

9% 

15% 

5 

11 

answered question 

46 

skipped question 

5 


3.2.6 ON THE IMPLEMENTATION OF THE “FOUR-STAGE MODEL” AND ESG 
FOUR-STAGE PROCESS FOR EXTERNAL QUALITY PROCEDURES 

The four-stage model is promoted as the shared foundation of European quality 
assurance and it has a prominent place in the criteria for ENQA membership. 

It is defined to be the following: (i) autonomy and independence in terms of 
procedures and methods concerning quality evaluation both from government and 
from institutions of higher education; (ii) self-assessment; (iii) external assessment by a 
peer-review group and site visits, and (iv) publication of a report. 

To explore compliance, first note that Question 24 deals explicitly with autonomy 
and most respondents reported their agencies to be either autonomous or “other” which 
in all but one case means that stakeholders are consulted. Combining the answers 
to Questions 29, 31 and 33 (46 agencies responded to all three) and deselecting those 
agencies where government has involvement in deciding the final outcome or in the 
formal appointment of the panel and selecting those which implement all four stages 
of the ESG four-stage process (see below) yields 22 out of 46 (47%) respondents. That 
means that less than 50% of the respondent agencies which addressed the question 
are fully compliant with the four-stage model [with 15 out of 25 (60%) full-member 
respondents non compliant]. 

The ESG four-stage process is different from the four-stage model. It is related to the 
four-stage model but describes four distinct processes explicitly and includes follow-up, 
viz. 


1. A self-assessment or equivalent procedure by the subject of the quality assurance 
process; 

2. An external assessment by a group of experts, including, as appropriate, student 
representation and site visits as decided by the agency; 

3. Publication of a report, including any decisions, recommendations or other formal 
outcomes; and 

4. A follow-up procedure to review actions taken by the subject of the quality 
assurance process in the light of any recommendations contained in the report. 

One respondent (ECA) indicated that the four-stage process question did not apply to 
it as it does not perform accreditation itself, only its members do. 29 out of 46 (63%) 
respondent agencies that carry out external quality procedures implement the four- 
stage process in full. 59% of Member/Candidate respondents implement all four stages 
and 64% of Associate/Afhliate respondents do (table 33). 

Twelve out of 32 (37.5% of) ENQA Full Member respondents do not implement all 
four components of the ESG four-stage process. All but one (86%) of the Candidate 
member respondents implements all four elements of the process. 

In one case where a published report is not used the respondent commented: “the 
decision is always published according to the national database” (ACQUIN); and 
another reported that what is published is a “statement of the applicant institution in 
terms of the expert reports. The report is only accessible [by] the assessed institution, 
and not published” (Austrian Accreditation Council). 

Table 33. 


WHICH OF THE FOLLOWING PROCESSES DOES YOUR AGENCY USE/ 
REQUIRE WITHIN ITS EXTERNAL QUALITY PROCEDURES? 

RESPONSE 

COUNT 

RESPONSE % 

A self-assessment procedure by the institution that is the subject of the 
external quality procedure 

45 

96 

An external assessment by a group of experts irrespective of whether or 
not there is a site visit 

42 

89 

A published report 

37 

79 

A follow-up procedure undertaken by the institution that is the subject 
of the external quality procedure in the light of any recommendations 
contained in the report 

34 

72 

Other (please specify) 

9 

19 

answered question 

47 

skipped question 

4 


One respondent commented that “a follow-up procedure is undertaken only in those 
cases when institution (or its programme) has not been granted full accreditation” 
(Lithuania). 

The component processes most commonly implemented by respondent agencies 
are self-assessment (96%) and external assessment by a group of experts (89%) (table 
33). While still commonly implemented, there is a notable fall-off in respondents’ 
implementation of the latter stages of the process. 

72% of agencies require a follow-up procedure to be undertaken by the institution 
that is the subject of the external quality procedure in light of any recommendations 
contained in the report. 


Underpinning the four stage process and in pole position in the four-stage model are 
autonomy (self-governance) and independence in terms of procedures and methods. 
The procedures for appointing experts and making the final decision on outcomes of 
external quality procedures are important indicators (among others) of independence. 
Table 34 presents the distribution of agents that formally appoint the external expert 
panel in respondent agencies. 

Table 34. 


WHO FORMALLY APPOINTS THE MEMBERS OF THE EXTERNAL EXPERT 
PANELS USED BY YOUR AGENCY? 

RESPONSE 

COUNT 

RESPONSE 

% 

Your Agency 

37 

80 

Other (please specify) 

9 

20 

Government (central/regional) 

4 

9 

The institution that is the subject of the external quality procedure 

0 

0 

answered question 

46 

skipped question 

5 


The question about who formally appoints the members of the external expert panels 
(table 34) may not have been clear to all respondents because a number did not select 
a given option but did make a comment. Nevertheless, it is clear that in most cases it is 
the agency that has the sole responsibility for appointing persons to the external expert 
panel (80%). 

Government has a role in a small proportion of cases (9%) but it is exclusive in only 
three cases. In no cases does the institution that is the subject of the external quality 
procedure formally appoint the external expert panels. Where neither the agency nor 
government formally appoints, this responsibility is likely to fall to an overarching 
accreditation or higher education body. Of the four agencies where the government 
makes the appointment two are ENQA members. In six cases, under the “other” 
category, bodies that are external to agencies formally appoint the members of the 
panels. 

The respondent agency is most likely to make the final decision on the outcomes 
of external quality procedures (74%) (table 35). In many cases where the respondent 
agency does not make the final decision the external panel does (in around 22% of 
cases). The government makes the final decision in around t3% of cases. There is some 
overlap in responses because different outcomes may have different decision makers. In 
87% of responses it is the agency or the expert panel that makes the final decision on 
the outcome of external quality procedures. 

In four cases where the respondent agency does not take the final decision it appears 
that it needs to report to some higher authority. 

Two respondents (AVEPRO, AQA) indicated that in some cases the higher education 
institution under review will make the final decision on the outcome of quality 
procedure. In both cases other sources are mentioned, such as government, quality 
assurance agency and national accreditation bodies. 


Table 35. 


WHO MAKES THE FINAL DECISION ON THE OUTCOME OF YOUR AGENCY'S 
EXTERNAL QUALITY PROCEDURES? 

RESPONSE 

COUNT 

RESPONSE % 


MEMBER AND 
CANDIDATE 

ASSOCIATE 
AND AFFILIATE 

Your Agency 

78% 

62% 

34 

74 

Other (please specify) 

25% 

23% 

11 

24 

The external expert panel 

28% 

8% 

10 

22 

Government (central/regional) 

13% 

15% 

6 

13 

The institution that is the subject of the 
external quality procedure 

3% 

8% 

2 

4 

answered question 

46 

skipped question 

5 


3.2.7 ON THE DIFFERENT AREAS ADDRESSED BY THE AGENCY’S EXTERNAL 
QUALITY PROCEDURES AT PROGRAMME LEVEL (WHERE APPROPRIATE) 
AND AT INSTITUTIONAL LEVEL AND THEIR RELATIVE IMPORTANCE 

Fourty three agencies responded to the part of the survey enquiring about the relative 
importance of areas addressed by their EQPs at programme level and at institutional 
level (research programmes leading to PhD, etc., if provided, is relevant here). 

Table 36 and figure 37 summarise the overall findings and in each case the options 
are sorted by average impact. 

At programme level, all respondent agencies who answered this part of the question 
(40) assess the professional and pedagogical qualifications of staff and consider it to be 
very important. It has the highest impact and rating. 

The five highest impacting items for programme level procedures are: “professional 
and pedagogical qualifications of staff”; “curriculum/syllabus”; “facilities and 
resources”; “internal quality assurance procedures”; and “mission/goals”. None of these 
items can be categorised as outcomes. 

Assessment of students including feedback to students is rated as important to very 
important by those who assess it but eight out of 40 (20%) of respondents do not assess 
this. 
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Table 36. Programme level 


INDICATE, USING THE DROP-DOWN MENUS, THE 
IMPORTANCE OF THE FOLLOWING WHERE THEY 
ARE ASSESSED BY YOUR AGENCY'S EXTERNAL 
QUALITY PROCEDURES AT PROGRAMME LEVEL [...] 

NOT ASSESSED 

UNIMPORTANT 

OF LITTLE 
IMPORTANCE 

MODERATELY 

IMPORTANT 

IMPORTANT 

VERY IMPORTANT 

RESPONSE COUNT 
NO. 

RATING AVERAGE 

IMPACT AVERAGE 

Professional and pedagogical qualifications of staff 

0 

0 

0 

1 

15 

24 

40 

4.6 

4.6 

Curriculum/syllabus 

2 

0 

0 

1 

9 

28 

40 

4.7 

4.5 

Facilities and resources 

1 

0 

0 

3 

18 

18 

40 

4.4 

4.3 

Internal quality assurance procedures 

2 

0 

0 

3 

14 

21 

40 

4.5 

4.3 

Mission/goals 

3 

0 

0 

5 

10 

22 

40 

4.5 

4.1 

Learning environments 

1 

0 

0 

6 

19 

14 

40 

4.2 

4.1 

Study structure 

4 

0 

0 

2 

12 

22 

40 

4.6 

4.1 

Teaching and learning methods 

2 

0 

2 

4 

15 

17 

40 

4.2 

4.0 

Feedback from students 

3 

0 

1 

5 

11 

20 

40 

4.4 

4.0 

Academic and personal support for students 

3 

0 

2 

1 

19 

15 

40 

4.3 

4.0 

Appropriateness of the learning outcomes attained 
by graduates 

4 

0 

0 

4 

15 

17 

40 

4.4 

3.9 

Student retention and completion rates 

1 

2 

2 

8 

11 

16 

40 

3.9 

3.9 

Connection between teaching and research 

4 

0 

1 

6 

11 

18 

40 

4.3 

3.9 

Employability of graduates 

5 

0 

0 

5 

17 

13 

40 

4.2 

3.7 

Research environment 

2 

2 

3 

10 

10 

13 

40 

3.8 

3.6 

Practical training periods (including placements in 
industry) 

4 

1 

2 

6 

19 

8 

40 

3.9 

3.5 

Assessment of students including feedback to 
students 

8 

0 

0 

4 

13 

15 

40 

4.3 

3.5 

Management, organisation 

5 

0 

1 

11 

11 

12 

40 

4.0 

3.5 

Internationalisation 

3 

1 

2 

11 

16 

7 

40 

3.7 

3.4 

Research output of staff and research students 

6 

0 

1 

11 

12 

10 

40 

3.9 

3.3 

Collaboration with other higher education 
institutions 

4 

0 

3 

15 

12 

5 

39 

3.5 

3.2 

Cooperation with other subjects, programmes 

6 

0 

2 

14 

11 

7 

40 

3.7 

3.1 

Research strategy 

8 

1 

1 

9 

16 

5 

40 

3.7 

3.0 

Research collaboration and links 

7 

0 

6 

10 

11 

6 

40 

3.5 

2.9 

Supervision of research students 

11 

0 

2 

6 

11 

10 

40 

4.0 

2.9 
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Figure 37. Programme Level 
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Appropriateness of the learning outcomes attained by graduates is in eleventh place on 
the list of 25 areas considered by programme-level procedures. 

At the institutional level (table 38 and figure 39), internal quality assurance 
procedures are considered to be from important to very important (with average rating 
of 4.7) and are assessed by 92% of the agencies. 

The five highest impacting items for institutional level procedures are: internal 
quality assurance procedures; management, organisation; mission/goals; professional 
and pedagogical qualifications of staff; and facilities and resources. 


Table 38. Institutional level 


INDICATE, USING THE DROP-DOWN MENUS, THE 
IMPORTANCE OF THE FOLLOWING WHERE THEY ARE 
ASSESSED BY YOUR AGENCY'S EXTERNAL QUALITY 
PROCEDURES AT [...] INSTITUTIONAL LEVEL [...] 

NOT ASSESSED 

UNIMPORTANT 

OF LITTLE 
IMPORTANCE 

MODERATELY 

IMPORTANT 

IMPORTANT 

VERY IMPORTANT 

RESPONSE COUNT 

RATING AVERAGE 

IMPACT AVERAGE 

Internal quality assurance procedures 

3 

0 

0 

3 

4 

27 

37 

4.7 

4.3 

Mission/goals 

4 

0 

0 

3 

8 

22 

37 

4.6 

4.1 

Management, organisation 

3 

0 

2 

2 

9 

21 

37 

4.4 

4.1 

Professional and pedagogical qualifications of staff 

4 

0 

0 

2 

12 

19 

37 

4.5 

4.0 

Facilities and resources 

3 

0 

1 

2 

15 

16 

37 

4.4 

4.0 

Feedback from students 

4 

0 

1 

4 

11 

16 

36 

4.3 

3.8 

Research environment 

3 

0 

2 

6 

13 

13 

37 

4.1 

3.8 

Research strategy 

5 

0 

0 

4 

14 

14 

37 

4.3 

3.7 

Academic and personal support for students 

3 

0 

3 

5 

13 

12 

36 

4.0 

3.7 

Learning environments 

4 

0 

1 

5 

18 

9 

37 

4.1 

3.6 

Connection between teaching and research 

4 

0 

2 

6 

14 

11 

37 

4.0 

3.6 

Employability of graduates 

5 

0 

0 

5 

17 

9 

36 

4.1 

3.6 

Internationalisation 

4 

0 

3 

4 

18 

8 

37 

3.9 

3.5 

Collaboration with other higher education institutions 

5 

0 

2 

7 

13 

10 

37 

4.0 

3.4 

Research output of staff and research students 

6 

0 

2 

5 

13 

11 

37 

4.1 

3.4 

Assessment of students including feedback to students 

6 

0 

4 

4 

9 

13 

36 

4.0 

3.4 

Research collaboration and links 

4 

0 

4 

8 

13 

7 

36 

3.7 

3.3 

Student retention and completion rates 

4 

1 

4 

8 

10 

9 

36 

3.7 

3.3 

Teaching and learning methods 

6 

2 

2 

6 

12 

8 

36 

3.7 

3.1 

Study structure 

8 

1 

1 

6 

11 

9 

36 

3.9 

3.1 

Appropriateness of the learning outcomes attained by 
graduates 

10 

0 

0 

8 

7 

11 

36 

4.1 

3.0 

Curriculum/syllabus 

9 

0 

3 

7 

9 

8 

36 

3.8 

2.9 

Practical training periods (including placements in 
industry) 

9 

1 

4 

7 

10 

4 

35 

3.5 

2.6 

Supervision of research students 

12 

0 

3 

5 

11 

6 

37 

3.8 

2.6 

Cooperation with other subjects, programmes 

8 

2 

4 

11 

8 

3 

36 

3.2 

2.5 


Figure 39. Institutional level 
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While the appropriateness of the learning outcomes attained by graduates is not 
assessed by institutional-level procedures in 10 cases out of 36 respondents (28%), 
where it is assessed it is considered important or very important. The employability of 
graduates (49%) and feedback from students (43%) are considered important assessment 
areas at the institutional level where assessed. 

At the institutional level 34 out of 37 respondents (93%) address at least one of 
the four research related areas and 24 out of 37 respondents (65%) address all four 
of them. Research strategy is considered as an important to very important part of 
institutional-level external quality procedures for 28 out of 37 respondents (75%). The 
research environment is also considered very important. Research output and research 
collaboration are considered somewhat less important. 26% of (9/34) respondents, who 
address research, do not address the “supervision of research students”. 

Two “other” areas that are assessed each by one respondent are “- development plan 
(five years) [which] is [considered] very important -[and] business plan to guarantee 
mid-term financial solvency [which] is [considered] very important” (Austrian 
Accreditation Council) and “Professional competence to work in the profession [which 
is considered] very important” (ECCE). 

Three of the comments under “other” were to explain that the question did not apply 
(ECA, Anonymous, German Accreditation Council). 

One respondent who completed the question made the point that “institutional 
evaluation has a more complex structure. Programme evaluation is undertaken only in 
an accredited institution and is less complex” (Russia). 

One of the respondents (CAQA Serbia) who marked the research criteria as 
important noted the agency shared responsibility for the QA of research activities: 

“there is parallel system for accreditation of research institutions and parts of 
educational institutions doing research activities. So it is important for CAQA to deal 
with research, particularly in terms of teacher’s qualification”. 

3. 2.7.1 Exploratory analysis 

The results in the tables seem to suggest that process and configuration criteria might 
be considered slightly more important than outcome criteria. This will be explored 
further. 

To aid interpretation of the data the areas have been classified into four domains viz. 
Process; Configuration; Outcome; and Goal. “Outcome” is concerned with evidence 
of attainment of educational goals (e.g. learning outcomes). “Goal” is concerned with 
targets, policy, plans and strategies. “Configuration” is concerned with descriptions 
and characteristics of the institution and its programmes. “Process” is concerned with 
how things work dynamically. Most areas can be discussed from all four perspectives 
but fit best with one. For example, research collaboration can be discussed and assessed 
in terms of linkages, facilitating processes, strategy and outcome but it fits best within 
configuration because it is more likely that the existence of collaborative links is the 
evidence that is most immediately relevant to an assessor under this heading. The 
strategic aspects, for example, will presumably be included in the institutional and 
programme strategies. 
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Table 40. 


CLASSIFICATION 

AREA ADDRESSED 

Configuration 

Collaboration with other higher education institutions 

Facilities and resources 

Research collaboration and links 

Learning environments 

Research environment 

Connection between teaching and research 

Professional and pedagogical qualifications of staff 

Study structure 

Curriculum/syllabus 

Cooperation with other subjects, programmes 
Management, organisation 
Internationalisation 

Goal 

Mission/goals 
Research strategy 

Outcome 

Student retention and completion rates 
Employability of graduates 
Feedback from students 

Research output of staff and research students 
Appropriateness of the learning outcomes attained by graduates 

Process 

Supervision of research students 
Internal quality assurance procedures 
Teaching and learning methods 

Assessment of students including feedback to students 
Practical training periods (including placements in industry) 
Academic and personal support for students 


To explore this further the average of individual ratings for each classification is 
considered. The average is over all members of the indicated respondent group (which 
could be All; Members and Candidates; and Affiliates and Associates) and over all 
the criteria in the category (Goal, Outcome, Configuration and Process). Repeating 
this calculation but replacing rating with impact yields metrics which emphasise the 
differences more but it does not significantly change the order. 

Table 41 reveals differences in average rating (Avg) but they are small in comparison 
with the standard deviations (SD). Goal-related criteria are rated as most important for 
institutional level external quality procedures followed by Outcome, Configuration and 
Process-related criteria in that order. Process and Outcome-related criteria are rated 
highest for programme-level procedures followed by Goal and Configuration-related 
criteria. There is very little difference in this pattern between the respondent groups. 


Table 41. Ratings analysis by respondent group and item category 



ALL RESPONDENTS 

AFFILIATES AND 
ASSOCIATES 

MEMBERS AND 
CANDIDATES 

AVG 

SD 

AVG 

SD 

AVG 

SD 

INSTITUTIONAL 

Goal 

4.4 

0.7 

4.5 

0.6 

4.4 

0.7 

Outcome 

4.1 

0.9 

4.1 

0.8 

4.0 

0.9 

Configuration 

4.0 

0.9 

4.1 

1.0 

4.0 

0.9 

Process 

4.0 

1.0 

4.2 

1.0 

3.9 

1.0 

PROGRAMME 

Goal 

4.1 

0.9 

4.2 

0.8 

4.1 

0.9 

Outcome 

4.2 

0.9 

4.2 

0.9 

4.2 

0.9 

Configuration 

4.1 

0.9 

4.0 

1.0 

4.1 

0.9 

Process 

4.2 

0.8 

4.3 

0.9 

4.2 

0.8 


Using a different approach, the mean of the average criteria rating/impact over each 
respondent group and category may be considered, this approach ignores variation 
in the number of responses to the different criteria. It turns out that the mean of 
the average ratings over each the two respondent groups over each category are not 
significantly different. Little additional insight is gained and the results are consistent 
with the previous ones. 

3.2.8 ON THE AGENCY’S INTERNAL QUALITY PROCEDURES 

Table 42 shows that a majority (89%) of respondent agencies have established 
procedures to ensure their own accountability. 

Internal quality assurance (IQA) is more likely to be implemented by Members and 
Candidates than Affiliates and Associates. Accountability procedures are established 
by 94% of ENQA Members and Candidates but by only 77% of Affiliates and Associates 
(one skipped this question). 

Three of the five respondent agencies that have not established accountability 
procedures indicated that it is because they are in the early stage of development and 
that they will be developed. 

Two agencies mention the use of ISO protocols and three indicate/give the 
impression that they are in the early stages of development of their IQA procedures: 
one (YODAK) reports that it has “some procedures which [are] currently under review”; 
another (EEAA) that the IQA procedures “are not fully established, but are being 
progressively put in place”; and the third one (ECCE) reports that it is addressing IQA 
“at present through its own Quality Assurance Committee although very much under 
development”. 


Table 42. 


HAS YOUR AGENCY ESTABLISHED PROCEDURES FOR ITS OWN 
ACCOUNTABILITY? 

RESPONSE 

COUNT 

RESPONSE % 


MEMBERS AND 
CANDIDATES 

ASSOCIATES AND 
AFFILIATES 

Yes 

94% 

77% 

41 

89 

No 



5 

11 

Comment 



17 


answered question 

46 

skipped question 

5 


The top five accountability measures are (table 43): 

• An enforced no-conflict-of-interest mechanism in the work of the agency’s 
external experts is established and enforced; 

• Mechanisms for collection and analysis of, and reflection on, feedback from 
experts and reviewed institutions for the purpose of informing and supporting the 
agency’s improvement and development; 

• Processes and results which reflect the agency’s mission and goals of quality 
assurance 

• An appeals system; 

• Internal quality assurance procedures which include an internal feedback 
mechanism and an internal reflection mechanism. 

An enforced no-conflict-of-interest mechanism for external experts is established and 
very effective in 42% of respondent agencies. It is the highest rated accountability 
measure. One respondent which does not have such a mechanism points out that its 
members do (ECA). Another indicated that “given our remit it is difficult to be specific 
as required above” (anonymous). 

Mechanisms for collection and analysis of, and reflection on, feedback from experts 
and reviewed institutions for the purpose of informing and supporting the agency’s 
improvement and development are established in all 43 respondents to this part and on 
average considered effective. 

Processes and results which reflect the agency’s mission and goals of quality 
assurance are widely implemented and considered effective, as would be expected, but 
only partly implemented in nine out of 41 (22%) cases. 

Thirteen out of 43 (30%) respondents to the question about an appeals system 
(including two ENQA members) either have not established or have only partly 
established such a system. Note that in Sweden “an appeals system is presently not 
possible due to the Swedish constitution. This will possibly be changed within the next 
two years.” 

Internal quality assurance procedures which include an internal feedback 
mechanism and an internal reflection mechanism are established in 65% of respondent 
agencies: 12% regard it as very effective and 44% as effective. Fifteen out of 43 (35%) 
respondents to this question either have not established or have only partly established 
such procedures. 


56 


Table 43. 


IN YOUR OPINION WHICH OF THE FOLLOWING 
ACCOUNTABILITY PROCEDURES ARE IN PLACE 
FOR YOUR AGENCY AND HOW EFFECTIVELY IS 
EACH FUNCTIONING? 

NOT ESTABLISHED 

PARTLY ESTABLISHED 

ESTABLISHED AND 
MODERATELY EFFECTIVE 

ESTABLISHED AND 
EFFECTIVE 

ESTABLISHED AND VERY 
EFFECTIVE 

MEMBERSAND 

CANDIDATES 

AFFILIATES AND 
ASSOCIATES 

RATING AVERAGE 

RESPONSE COUNT 

An enforced no-conflict-of-interest mechanism 
in the work of your Agency’s external experts is 
established and enforced 

2 

5 

4 

14 

18 

4.4 

2.9 

4.0 

43 

Mechanisms for collection and analysis of, and 
reflection on, feedback from experts and reviewed 
institutions for the purpose of informing and 
supporting your Agency’s improvement and 
development 

0 

9 

7 

17 

10 

3.9 

3.1 

3.7 

43 

Processes and results which reflect your Agency’s 
mission and goals of quality assurance 

1 

9 

4 

19 

8 

3.8 

3.1 

3.6 

41 

An appeals system 

6 

7 

6 

15 

9 

3.4 

3.2 

3.3 

43 

Internal quality assurance procedures which 
include an internal feedback mechanism and an 
internal reflection mechanism 

4 

11 

4 

19 

5 

3.6 

2.4 

3.2 

43 

Mandatory cyclical external review of your 
Agency’s activities at least once every five years 

9 

8 

3 

11 

10 

3.5 

2.2 

3.1 

41 

A published policy for the assurance of your 
Agency’s own quality, made available on its 
website 

9 

8 

6 

11 

9 

3.4 

2.3 

3.1 

43 

Provision for periodic independent evaluation and 
reporting of your Agency’s conformity with the 
membership criteria of ENQA. 

10 

5 

10 

9 

7 

3.5 

1.8 

3.0 

41 

Reliable mechanisms that ensure the quality 
of any activities and material produced by 
subcontractors, if some or all of the elements in 
your Agency’s quality assurance procedure are 
subcontracted to other parties 

11 

3 

7 

10 

2 

2.8 

2.3 

2.7 

33 

Other (please specify) 









9 

answered question 

43 

skipped question 

8 


Mandatory cyclical external review of the agency’s activities at least once every five 
years is not established in nine out of 41 (22%) respondent agencies including at least 
(some skipped the question) three ENQA full member agencies. However, the full 
member agencies have time until the end of 2010 to undergo the external review. 

To the question about having reliable mechanisms that ensure the quality of any 
activities and material produced by subcontractors, if some or all of the elements in 
an agency’s quality assurance procedure are subcontracted to other parties, 14 out 
of 33 (43%) respondents have either not established or only partly established such a 
mechanism. However, there is a possibility that some respondents answered this in the 
negative where they should have skipped it entirely if inapplicable. 


Provision for the periodic independent evaluation and reporting of the agency’s 
conformity with the membership criteria of ENQA is absent in to out of 41 (24%) 
responses, including two ENQA members. 

One interesting addition to the list of accountability measures listed in the 
questionnaire is the “publication of all the external expert panel reports with the CVs 
of the experts and their undersigned statement of independence and confidentiality” 
(NVAO). 

Two respondents stress that their accountability procedures are new. 

3.3 Main topics for interpretation 

3.3.1 ANALYSIS OF AGENCY FUNCTIONS, OBJECTIVES AND PRIORITIES 
AND THE INVOLVEMENT OF STAKEHOLDERS IN ITS WORK 

The European Standards and Guidelines document places a strong emphasis on the 
interests of students and other stakeholders, such as labour market representatives, 
which “should be at the forefront of external quality processes”. The involvement of 
stakeholders is an indicator of higher education’s shift from a teaching to a learning 
focus, its relationship to society, industry, and commerce, to employers and professional 
needs, and its interest in international comparability. Stakeholder involvement can 
enhance the transparency and confidence in external quality procedures and the 
standing of the higher education sector as a whole. 

However, the ESG also make it clear that it is important that the external quality 
assurance agency is autonomous and that its procedures are independent: “Agencies 
should be independent to the extent both that they have autonomous responsibility for 
their operations and that the conclusions and recommendations made in their reports 
cannot be influenced by third parties such as higher education institutions, ministries 
or other stakeholders.” 

Most respondent agencies have a board/council (table 23) and more than half (57%) 
of respondents has representation from four or more stakeholder groups (table 24). 

There have been some significant increases in stakeholder representation on boards/ 
councils from 2003. Notably, there has been an increase in student representation (now 
49%) on respondent agency boards/councils since 2003, when only a third of boards 
had student members. While an improvement, the current level still seems low in the 
context of a recognised need for greater emphasis on how higher education affects 
students. The proportion of boards/councils with representation of higher education 
institutions is 82% which is also a significant increase on 62% in 2003. Slightly less 
than one third of respondent agency boards (31%) (table 24) include an international 
quality assurance and/or higher education perspective. This limited representation by 
international quality assurance or higher education bodies is noteworthy, given that 
many have transparency as an objective and given that international comparability and 
peer review are priorities under the Bologna process. 

Moving from stakeholder involvement to agency functions: in 2003 the main agency 
functions were reported as being: (i) quality improvement, and quality assurance in a 
traditional sense; (ii) disseminating knowledge and information; and (iii) accreditation. 
The current survey found the five functions that have the highest impact to be “quality 
assurance”; “quality enhancement/improvement”; “external quality assurance of 
programmes”; “external quality assurance of institutions”; “collecting/ disseminating 
information on quality of HE” and most respondents have all of those as functions. The 


2008 survey results seem broadly consistent with the 2003 survey findings (but recall 
that the questions are different). 

Many respondent agencies have other complementary functions but these are 
considered to be of somewhat lower importance to the agency’s mission than the 
main ones (table 26). Significantly, 27 out of 43 respondents see the “development and 
maintenance of qualifications frameworks” as a one of their functions and among those 
its average rating is more than important (4.11). This is a very significant increase on 
the three agencies that reported this as a task in the 2003 report. It is an expected 
finding owing to the Framework for Qualifications of the European Higher Education 
Area (FQEHEA) and the European Qualifications Framework (EQF). Recognition 
of national diplomas is another function where there has been a large increase from 
one in 2003 to 20 in 2007. Removing barriers to mobility is an objective of the 
Bologna process. This increase suggests a strong linkage between recognition and 
external quality assurance which is understandable. Such a linkage can be expected to 
motivate a growing interest in collaborative/standardised approaches including mutual 
recognition agreements between quality assurance agencies. 

The objectives of the main types of respondent agencies’ quality assurance 
procedures are broadly consistent with the stated functions. The top five agency 
objectives, in order of impact, are “transparency”, “quality improvement/enhancement”, 
“accountability”, “accreditation” and “student (or prospective student) protection” 

(table 28). In 2003 Quality improvement was the predominant objective followed by 
accountability and transparency (in joint second place). Given that transparency is the 
top objective and given that stakeholder involvement has increased transparency, it 
is curious how 51% of agencies can seem to justify not having student representation 
on their boards/councils. To better interpret this it would be necessary to ask agencies 
what they understand by the objectives selected. 

Fourteen percent (six out of 44) of respondents consider the estimation of graduate 
learning outcomes not to be an objective of their procedures. Most of those who have it 
as an objective, consider it to be important to very important. 

The objectives associated with making national and international comparisons are 
on average rated lowest but for ranking, by respondents. However, the proportion of 
respondent agencies involved in these activities is about 80%, which is significantly 
higher than in the 2003 sample. Notwithstanding this, the relatively low rating/impact 
may warrant further investigation given that comparability and compatibility of higher 
education institutions is a key theme in the Bologna process. 

More than half of the respondents do not have rankings as an objective and, where 
it is and objective, it is on average considered to be unimportant to of little importance. 
This is consistent with the 2003 findings (table 28). 

Finally, one particularly noteworthy objective reported by the QAA, UK relates 
to ensuring that regulation is appropriate to the level of assurance sought. This is an 
objective that might usefully be adopted by all agencies which do not already include it. 

3.3.2 ANALYSIS OF THE IMPLEMENTATION OF THE ‘FOUR-STAGE MODEL’ 
AND VARIANTS OR ALTERNATIVES 

Responses to the 2007 survey demonstrate that the four-stage model is frequently 
(nearly 50%) but not universally implemented. 


Compliance with the (distinct) ESG four-stage process is higher at 29 out of 46 
(63%) (ECA responding indicated that the ESG four-stage model did not apply to its 
operations). 59% of Member/Candidate respondents (52% of Full members and 86% 
of Candidate members) implement all four ESG stages and 64% of Associate/Affiliate 
respondents do. A published report is required by 79% of agencies which is lower than 
the 2003 finding (when 87% of respondent agencies published a report) (table 33). 

Of the respondent agencies who implement the ESG four-stage process fully 59% 
involve student representatives in a formal role in the specification of processes and 
criteria; 55% have student representation on their board/council; and 90% have 
established accountability procedures. All this may suggest that implementation of the 
model may be correlated with a progressive overall approach. 

Even where respondents claim to implement the ESG four-stage process in full, there 
are instances where not all four stages are considered necessary. 

That the four stage model is not being implemented fully among so many ENQA 
members gives pause for thought. The relatively high non-compliance (given the age 
of the model) could be due to the relatively long lead time for necessary changes in 
legislation to be implemented or it could be that the reality is more complex than the 
model. Both of these notions were discussed in Chapter 3.2. 

The non-compliance with the four-stage model arises from government involvement 
and lack of implementation of the latter stages of the ESG four-stage process. However, 
as noted in Chapter 3.2 the inference of independence from government is not 
straightforward. Consider that government is represented on 40% of boards/councils 
(table 23). 

It is possible to speculate about the causes for the apparent level of resistance to 
this model. Publication of reports is good for transparency but challenges candour and 
increases the likelihood of anodyne commentary. Independence from government and 
institutions is a stark criterion however it is glossed. It is important that external quality 
procedures should bring a perspective that is different from the institution, free from 
conflict of interest and objective in basing its decision exclusively on material evidence. 
Obviously, independence is not sufficient to provide this perspective and clearly some 
feel that absolute independence it is not necessary. 

3.3.3 ANALYSIS OF THE AREAS ADDRESSED BY EXTERNAL QUALITY 
PROCEDURES 

The five highest impacting items assessed by programme level procedures are: 
“professional and pedagogical qualifications of staff”; “curriculum/syllabus”; “facilities 
and resources”; “internal quality assurance procedures”; and “mission/goals”. None of 
these items can be categorised as outcomes (table 36 and figure 37). The five highest 
impacting items for institutional level procedures are: internal quality assurance 
procedures; management, organisation; mission/goals; professional and pedagogical 
qualifications of staff; and facilities and resources (table 38 and figure 39). 

The ESG states that “the assessment of students is one of the most important 
elements of higher education”. Assessment of students links the ESG with the 
FQEHEA. Given the focus at policy level on learning outcomes (exemplified by the 
two qualifications frameworks) learning-outcome centred areas such as assessment 
of students including feedback and appropriateness of learning outcomes attained 
by students, etc. are areas that should not be overlooked by agencies. Assessment of 


students including feedback to students is rated as important to very important by 
those who assess it but eight out of 40 (20%) respondents do not assess it. Indeed, 
appropriateness of the learning outcomes attained by graduates is in eleventh 
place on the list of 25 areas considered by programme-level procedures. While the 
appropriateness of the learning outcomes attained by graduates is not assessed in 10 
out of 36 (28%) respondents’ institutional-level procedures, where it is assessed it is 
considered important or very important. 

Research assessment is addressed by most respondents to this part (the institutional- 
level procedures). Thirty four out of 37 (93%) respondents address at least one of the 
four research-related areas and 24 out of 37 (65%) respondents address all four of them. 
Research strategy is an important to very important part of institutional-level external 
quality procedures for 28 out of 37 (75%) respondents. The research environment is also 
considered very important. Research output and research collaboration are considered 
somewhat less important. 26% of (9/34) respondents, who address research, do not 
address the “supervision of research students”. 

Finally, if the areas are classified into Goal, Outcome, Process and Configuration 
(table 40) then Goal-related criteria are rated as most important for institutional level 
external quality procedures followed by Outcome, Configuration and Process-related 
criteria in that order; (table 41) and Process and Outcome-related criteria are rated 
highest for programme-level procedures followed by Goal and Configuration-related 
criteria. There is very little difference in this pattern between the respondent groups. 

3.3.4 ANALYSIS OF INTERNAL QUALITY PROCEDURES 

The European Standards and Guidelines state that agencies should have in place 
procedures for their own accountability. These procedures are to include: a published 
policy, supporting documentation and a mandatory cyclical external review at least 
once every five years 13 . 89% of respondents (41/46) claim to have procedures for their 
own accountability. The top five accountability measures are (table 43): 

• A no-conflict-of-interest mechanism in the work of the agency’s external experts is 
established and enforced; 

• Mechanisms for collection and analysis of, and reflection on, feedback from 
experts and reviewed institutions for the purpose of informing and supporting the 
agency’s improvement and development; 

• Processes and results which reflect the agency’s mission and goals of quality 
assurance 

• An appeals system; 

• Internal quality assurance procedures which include an internal feedback 
mechanism and an internal reflection mechanism. 

In four of the top five accountability procedures the average effectiveness is between 
“moderately effective” and “effective” (table 43). Evidently, agencies see potential 
for improvement in many areas. Only five out of 28 (18%) respondents which have 
established internal quality assurance procedures report then as being very effective. 

Overall ENQA Members/Candidates perform significantly better than Affiliates/ 
Associates on accountability criteria. 


ESG, p.26 


Notwithstanding this, when it comes to mandatory cyclical external review of the 
agency’s activities at least once every five years, two ENQA members skipped this and 
three have not yet established it (but have still time until the end of 2010). 

Finally, reliable mechanisms that ensure the quality of any activities and material 
produced by subcontractors, if some or all of the elements in an agency’s quality 
assurance procedure are subcontracted to other parties may be a significant weakness. 


Chapter 4: Division of responsibilities 
and methodological framework of 
procedures 

4.1 Introduction 

This section of the report deals with the following areas: 

• Membership of expert panels and their training; 

• Division of function and responsibility in external quality procedures; 

• Reference frame of, and criteria and standards for external quality procedures; 
and 

• Procedures for the external quality assurance of collaborative provision. 

The section deals with each of the above topics in turn, and sets out a description of the 
statistical responses received, along with the presentation of aspects of the qualitative 
responses, where relevant. This is followed by a brief comparison with similar aspects 
of the last Quality Procedures Survey, which was carried out in 2003. 

Several points should be borne in mind when reading this section of the report. 
Firstly, the (relatively) abnormal number of developments that has taken place since 
the last survey in 2003; for example, the publication of the Standards and Guidelines for 
Quality Assurance in the EHEA, the Framework for Higher Education Qualifications in the 
EHEA, etc. has meant that there has been an enormous amount of activity in this field 
in the last five years; much of the activity might certainly indicate trends in particular 
areas, but it may not yet be well-embedded in the systems of any particular national 
context. 

There are several areas (the training of experts, the reference frame for external 
evaluations and quality assurance of collaborative provision) which were not covered in 
the 2003 survey. Comments on those particular areas will, therefore, be set out for the 
first time. 

4.2 Membership of expert panels and their training 

4.2.1 MEMBERSHIP 

Table 44. 


WHO ARE THE MEMBERS OF THE EXTERNAL EXPERT PANEL? 

RESPONSE 

COUNT 

RESPONSE % 

National experts representing area of focus 

35 

76.1 

International experts 

34 

73.9 

Professional practitioners 

29 

63.0 

Students 

26 

56.5 

National experts representing institutions 

15 

32.6 

Employers 

21 

45.7 

Staff members of the quality assurance agency 

12 

26.1 

Other (please specify) 

12 

26.1 

Professional organisations 

9 

19.6 

Graduates 

7 

15.2 

answered question 

43 

skipped question 

8 


A majority of processes (76%) make use of national experts representing the area 
of focus (table 44). In fact, this majority may be even larger due to the fact that the 
questionnaire separated “area of focus” from experts who represented institutions. 

In fact, if the area of focus is the institution, then such experts could be added to the 
majority. A high percentage of respondents indicated that the composition of their 
expert panels included international representation (74%) and professional practitioners 
(63%). The number who indicated student participation in external expert panels 
was lower (56%, but see paragraph below on comparison with 2003). There is no 
real differentiation between the composition of panels and the type of agency - for 
example, an accreditation agency with a specific subject focus does not differ radically 
in the composition of its panels to other types of agencies. 

Some of the commentary in response to this question indicates that students and 
international experts will be represented from 2008. 

4.2.2 TRAINING 

The majority of agencies (83%) offer some kind of training or briefing to external expert 
teams. The questionnaire did not distinguish between training and briefing, and it is 
evident that the two activities can in some cases be considered quite different. From the 
answers it could be deduced that such training/briefing lasts, on average, for 1.5 days. 
Responses ranged from two hours to five days. 

75% of respondents indicated that training was compulsory, with 25% saying that an 
expert could not be appointed to a panel until they had completed training. 22.5% of 
agencies said that training might be provided by means of distance learning (table 45). 
There is evidence that these agencies use international panel members, suggesting that 
some form of distance learning is considered as a cost-effective and convenient means 
of briefing such panel members. 

It is interesting to note that a relatively high numbers of agencies skipped this 
question. 

Table 45. 


CHARACTERISTICS AND CIRCUMSTANCES OF BRIEFING/TRAINING 
PROVIDED TO PANEL MEMBERS 

RESPONSE 

COUNT 

RESPONSE % 

Training/briefing is compulsory for all panel members 

30 

75 

Training/briefing is compulsory for panel chairs 

20 

50 

Training/briefing provided is matched to the experience of each panel 
member 

17 

42.5 

Training outcomes are assessed by my Agency 

11 

27.5 

Training is required before a person may be nominated to a panel 

10 

25 

Training is provided using distance learning 

9 

22.5 

Other (please specify) 

7 

17.5 

answered question 

40 

skipped question 

11 


The content of any training/briefing sessions generally focuses on the methodology 
to be used and the function of the panel arising out of that methodology. However, 
probably dependent on whether the event was viewed more as training or as briefing, 
there were significant variations in responses. For example, some sessions are intended 


to provide guidelines to panels or to point them in the direction of good practice. 

Others focus on training the team to carry out the task and provide training tools 
such as learning logs so that the experts can apply what they have learned during the 
procedure itself. Several responses indicated that it was necessary to inform experts 
of the legal situation in which they would be working, and many of the respondents 
indicated that an explanation of the education system, alongside any relevant legislation 
formed part of the training/briefing. It is possible that such aspects of training could 
be mapped to those agencies which use international experts as part of their teams 
or those agencies that work in a context where programmes and/or institutions are 
accredited or approved by the state. Only three respondents referred to the ESG and 
the Bologna Process. In some cases, analysis of the self-evaluation document and 
preparation for the site visit was considered to be training or briefing, whereas for 
other agencies, these were activities that are carried out individually by panels/panel 
members. 

One agency pointed out that it did not offer any training or briefing because it had 
an oversight role and was therefore not involved at this level of detail. (This comment 
can be taken as read for responses to other questions in this chapter of the survey). 

4.2.3 COMPARISON WITH THE 2003 SURVEY 

In terms of direct comparisons with the figures produced, the percentage of national 
experts representing the area of focus and the percentage of international experts on 
panels has remained the same. The number of student members has risen significantly 
(from 22% in 2003). Given the level of recent debate on the matter of student 
involvement in quality assurance, this is not surprising. If the category of “Employers” 
in the 2003 survey can be assumed to be equivalent to “professional practitioners” then 
this percentage too has risen significantly (from 36% in 2003). The number of agencies 
where staff is cited as being represented on external panels has dropped from 40% to 
26%. 

Training of external experts was not covered in the 2003 survey. 

4.3 Division of function and responsibility in external QA procedures 

4.3.1 DIVISION OF FUNCTION 

In terms of the choice and development of any external quality procedures, all 
responding agencies performed this function (table 46). In about a quarter of cases, the 
external expert panel was also involved. A very high percentage of agencies (between 
84% and 98%) were involved in the development of guidelines for self-evaluation, 
contact with the institution, and preparation of the site visit, with involvement of the 
expert panel in such functions ranging from 9% (preparation of guidelines for self- 
evaluation) to 52% (planning of the site visit). 

The statistics are significantly reversed when the report-writing function is 
considered - 94% of expert panels are responsible for writing the report, compared 
to only 37% of agencies, and, in fact, when the responses are considered in detail, 
the majority of this 37% say that they carry out this function jointly with their expert 
panels. 


Table 46. 


WHO PERFORMS THE FOLLOWING FUNCTIONS IN THE 
EXTERNAL QUALITY PROCEDURES (IF THE FUNCTIONS ARE 
PERFORMED IN COOPERATION BETWEEN THE EXTERNAL 
EXPERT PANEL AND THE QUALITY ASSURANCE AGENCY, 
PLEASE MARK BOTH) 

THE 

EXTERNAL 

EXPERT 

PANEL 

THE QUALITY 
ASSURANCE 
AGENCY 

RESPONSE 

COUNT 

Choice of basic methodology applied 

12 

44 

44 

Preparation of the guidelines for the self-evaluation 

4 

42 

43 

Preparation of the external quality procedure concept 

11 

42 

42 

Contact with the institution 

11 

41 

44 

Planning of the site visit 

23 

36 

43 

Preparation of the guidelines for the site-visit 

16 

38 

42 

Writing of the report 

41 

17 

44 

Other 

1 

2 

3 

Other (please specify) 



6 

answered question 

44 

skipped question 

7 


4.3.2 RESPONSIBILITY FOR THE DIFFERENT PARTS OF THE EXTERNAL 
QUALITY PROCEDURE 

There is a marked division in the responses to this question (table 47): the description 
and analysis of any process being externally evaluated, conclusions drawn and 
recommendations made are quite clearly the responsibility of the expert panel (88- 
96% of responses). However, this situation is reversed in terms of responsibility for the 
decision about the outcome of the external evaluation process and the preparation of 
the report for publication (74% and 84% of agencies saying that they were responsible 
for these respective parts of the process). It may, however, be worth remembering here 
that whilst some agencies might not take the final decision on the outcome, nonetheless 
the outcome and its subsequent reporting are owned by the agency. It was also pointed 
out that a final accreditation decision or judgement would be owned by the agency. 
Thirty agencies of diverse status and purpose stated that they owned the formal 
outcome or decision from an evaluation. 


Table 47. 


WHO IS RESPONSIBLE FOR THE DIFFERENT PARTS OF THE 
EXTERNAL QUALITY PROCEDURES? (IF BOTH THE EXTERNAL 
EXPERT PANEL AND THE QUALITY ASSURANCE AGENCY 
ARE RESPONSIBLE, PLEASE MARK BOTH) 

THE 

EXTERNAL 

EXPERT 

PANEL 

THE QUALITY 
ASSURANCE 
AGENCY 

RESPONSE 

COUNT 

The description and analysis 

38 

17 

43 

The conclusions 

42 

11 

44 

The recommendations 

40 

17 

44 

The formal decision (evaluation outcome) 

15 

30 

40 

The report prepared for publication 

19 

35 

42 

Other 

1 

2 

2 

Other (please specify) 



8 

answered question 

44 

skipped question 

7 


4.3.3 COMPARISON WITH THE 2003 SURVEY 

The results of the 2003 survey are similar in terms of the responsibility of the experts 
with regard to description, analysis, conclusions and recommendations, although this 
responsibility appears to have increased a little between 2003 and 2007/08. 

The 2003 survey does not consider the responsibility of the decision regarding the 
outcome of the external process. 

4.4 Reference frame of, and criteria and standards for external quality 
procedures 

4.4.1 THE FRAME OF REFERENCE FOR THE EXTERNAL QUALITY 
PROCEDURE 

The following were offered by the survey as potential frames of reference for any 
external quality assurance procedure: the goals of the institution undergoing 
evaluation, guidelines for good practice, legal regulation, a national framework for 
qualifications, the EHEA framework for qualifications, subject benchmarks and 
standards set by professional organisations. 

Table 48. 


WHAT DEFINES THE REFERENCE FRAME FOR THE EXTERNAL QUALITY 
PROCEDURE? 

RESPONSE 

COUNT 

RESPONSE % 

Legal regulation 

35 

74.5 

The stated goals of the institution that is the subject of the external 
quality procedure 

27 

57.4 

Guidelines of good practice 

27 

57.4 

National Framework of/for Qualifications 

26 

55.3 

A Framework for Qualifications of the EHEA 

21 

44.7 

Standards defined by professional organisations 

16 

34 

Subject benchmarks 

13 

27.7 

Other (please specify) 

12 

25.5 

answered question 

47 

skipped question 

4 


74.5% of respondents said that their frame of reference was defined by legal regulation 
(table 48). These agencies were mostly those based in Eastern Europe and/or are 
accreditation agencies. However, it should be noted that none cited legal regulation as 
the sole basis of their frame of reference. 

There was a cluster of responses (43%-57%) around qualification frameworks (both 
national and EHEA), the stated goals of the institution and guidelines for good practice. 
It should be noted however, that some respondents specified that their national 
framework of qualifications was still under development. The lowest responses were for 
professional standards (34%) and subject benchmarks (27.7%). 

More detailed responses indicated that members of the European Accreditation 
Consortium (ECA) were committed to ECA’s Code of Good Practice, whilst others 
mentioned the ESG specifically, although it is possible that others will have assumed 
these to have been covered by the ‘guidelines for good practice’ category. 

4.4.2 SPECIFIC CRITERIA AND STANDARDS 

Respondents indicated that the main sources of specific criteria and standards were 
their own (87%) and the ESG (83%). National criteria and standards were also prevalent 


(66%). The survey also offered the option of the OECD/UNESCO guidelines on quality 
provision in cross-border HE. Only 14% of respondents indicated that these formed part 
of their criteria and standards for external procedures. 

Again, the ECA Code of Good Practice was mentioned (although not by all member 
agencies of ECA) and also the guidelines published by the World Health Organisation 
(one response). 

Table 49. 


WHAT SPECIFIC CRITERIA AND STANDARDS ARE USED FOR YOUR 
AGENCY'S EXTERNAL QUALITY PROCEDURES? 

RESPONSE 

COUNT 

RESPONSE % 

The Agency’s own published criteria and standards 

41 

87.2 

European Standards and Guidelines 

39 

83 

National criteria and standards 

31 

66 

OECD/UNESCO Guidelines on Quality Provision in Cross-border 
Higher Education 

7 

14.9 

Others (please specify all and provide url links) 

5 

10.6 

answered question 

47 

skipped question 

4 


4.4.3 COMPARISON WITH THE 2003 SURVEY 

The 2003 survey describes the notion of criteria and standards as “emerging as a 
common feature” and as being much more in evidence than in 1998. The results of the 
2007-08 survey demonstrate that this feature could now be described as embedded. 
However, the 2003 survey states that criteria and standards were emerging beyond 
previous terms of reference which were almost entirely based on legal regulations. The 
2007-08 survey separates the framework of reference and the specific criteria and 
standards and, as indicated above, legal regulations still account for the majority of the 
responses in terms of the framework of reference. Care must be taken, therefore, in 
drawing conclusions. 

Indeed, the answers to many of the questions raised in the 2003 survey still cannot 
be answered definitively [e.g. ‘what is the difference between criteria and standards?’ 
(In the case of this question, it would be interesting to know if the development of 
the ESG has helped to clarify this point), ‘when does an agency work with threshold 
criteria?’ ‘Is it important whether the criteria are explicitly formulated or not?’]. 
However, the current survey does seek to provide answers to other questions posed in 
2003, e.g. ‘who formulates the criteria?’ 

The publication of the ESG will also impact on the comparisons that one can draw 
in this area; it is possible that some newer agencies will have adopted the ESG without 
having developed their own at any time due to the circumstance of timing, or, indeed, 
that the ESG were published as some of the more well-established agencies were 
reviewing their own activities. 

4.5 Quality assurance of collaborative provision 

4.5.1 THE EXISTENCE OF EXTERNAL QUALITY PROCEDURES FOR JOINTLY/ 
COLLABORATIVELY PROVIDED PROGRAMMES DELIVERED BY A GROUP OF 
INSTITUTIONS IN ONE COUNTRY 

Just over half (56.5%) of the responding agencies said that they had procedures 
specifically for the evaluation of collaborative provision. However, care should be taken 


in making assumptions around this figure as no definition of the term “collaborative” 
or “joint” was provided. A couple of agencies that had provided a qualitative response 
stated that same procedures were used for collaborative/joint provision as for other 
provision. 

4.5.2 THE EXISTENCE OF EXTERNAL QUALITY PROCEDURES FOR JOINTLY/ 
COLLABORATIVELY PROVIDED PROGRAMMES THAT ARE DELIVERED 
ACROSS NATIONAL BOUNDARIES 

Again, just over half (55%) of responding agencies said that such procedures for 
programmes delivered across national boundaries were in place. Qualitative responses 
indicated that, in some cases, procedures are to be developed in the near future, 
particularly to deal with the development of Erasmus Mundus joint programmes. One 
response stated that it would expect any partnerships to adhere to the ESG. The ECA 
guidelines were also cited. Other responses indicated that transnational provision 
was dealt with on a case by case basis or that standard QA procedures were adapted 
accordingly. 

4.5.3 COMPARISON WITH THE 2003 SURVEY 

This area was not covered in the 2003 survey - it would be interesting to note trends 
and developments in the external quality assurance of collaborative provision in any 
future survey of this type. The proliferation of programmes such as Erasmus Mundus 
joint masters is sure to mean further development in this area in the near future. 

4.6 Conclusion 

Readers may be interested to compare the findings of this chapter with those of 
Chapters 2 and 3. Of particular note is the fact that many agencies apply more than one 
type of external quality assurance, although the four-stage model (see 3.2.6) is the most 
common throughout. It might also be surmised that the publication of the ESG has 
helped agencies to clarify their purpose. 

A further interesting study would be to compare the outcomes above with the 
conclusions of the Quality Convergence Studies I and II;( Crazier, Curvale & Henard, 
ENQA, March 2005 and March 2007); how much variation is there? And, in the cases 
of variation, are agencies aiming for the same outcome but via different methods? Or 
are they looking to achieve different outcomes with similar methodologies? 


Chapter 5: Site visits 

The characteristics of the quality assurance systems, mentioned in the Council 
Recommendation of 24 September 1998 on European Co-operation in Quality 
Assurance in Higher Education, correspond to the four-stage model: independent 
agencies, self-evaluations, visits by experts and a published report. These were 
identified as common features of European quality assurance in the European Pilot 
Project for Evaluating Quality in Higher Education of 1995 and in Evaluation of European 
Higher Education: A Status Report ofigg8. The four-stage model is widely accepted as 
the common ground for the European quality assurance in higher education. As such it 
has become the criteria for ENQA membership. 

Questions 50-56 of the survey addressed institutional self-evaluation as well as the 
site visit and directly refer to the four-stage model and ESG. 

The findings are discussed in the following and commonalities in self-evaluation 
procedures are identified. 

5.1 Institutions' self-evaluation groups 

The capacity for objective self-evaluation is necessary for both professional and 
institutional development. It is the staff of an institution (academic, administrative, 
governing, etc) who are principally responsible for quality. The constitution of self- 
evaluation groups is a critical factor in their success or otherwise. 

Agencies participating in the survey were asked to indicate who is included in the 
self-evaluation groups of the institutions that were subject to their quality assurance 
procedures. The overall findings in table 50 show that a majority of respondents 
listed teaching staff and management as the most commonly included categories. 
Administrative staff and students came second. Graduates were rarely included in 
the institutions’ self-evaluation groups. A self-evaluation group was not used in three 
cases: HETAC (Ireland), CNVSU (Italy) and an anonymous agency. In the cases of ECA 
(varies per member), German Accreditation Council (only accredits agencies) and an 
anonymous respondent (oversight role of the agency) this question was not applicable. 

Overall, and compared with the previous survey (2003), the responses showed no 
difference in the most frequently included group (management and teaching staff) and 
in the least often included group (graduates). External stakeholders were mentioned 
once, in the “other” category. 

One surprising finding is the low number of agencies that report that institutions’ 
self-evaluation includes persons in the ‘other’ category. In particular, this implies that 
the number of institutions using external experts (discipline experts, facilitators, etc) in 
the self-evaluation is low. Respondents were not prompted to consider this category and 
it is possible that if they were, the number would have been higher. 


Table 50. 


WHO, TYPICALLY, IS INCLUDED IN THE INSTITUTIONS' SELF 
EVALUATION GROUP? 

RESPONSE 

COUNT 

RESPONSE % 

Management 

39 

84.8 

Teaching staff 

39 

84.8 

Students 

33 

71.7 

Administrative staff 

33 

71.7 

Graduates 

13 

28.3 

Self-evaluation group is not used 

3 

6.5 

Other 

6 

13 

answered question 

46 

skipped question 

5 


5.2 Sources for data collection 

Agencies were further asked to identify what kind of statistical data was used in the 
institutions’ self-evaluation. All respondents indicated that they were using statistical 
data. The most frequently occurring type of data was “data on students”; followed by 
“data on teaching staff”; “data on administrative staff” and “financial key figures” (see 
table 51). Just as in the 2003 survey, labour market statistics ranked lowest. 

The collection of employers feedback can provide evidence on the quality of 
graduates’ learning outcomes, and consequently on the quality of the institution. 
Therefore it maybe expected that the trend of conducting regular employer surveys 
might get stronger in the future. The analysis of responses under ‘other’, however, 
revealed a variety of data sources, for example: curriculum, syllabus, student workload, 
research outcomes and publications, facilities and equipment, development plan and 
mobility. 

Table 51. 


WHAT SORT OF STATISTICAL DATA IS USED IN THE SELF-EVALUATION? 

RESPONSE 

COUNT 

RESPONSE % 

Data on students (e.g. in take, drop-out rate, passing rate) 

43 

93.5 

Data on teaching staff 

42 

91.3 

Data on administrative staff 

34 

73.9 

Financial key figures 

27 

58.7 

Labour market statistics 

26 

56.5 

Other 

13 

28.3 

Statistical data is not used 

0 

0 

answered question 

46 

skipped question 

5 


5.3 Site visits 

The site visit is conducted, in the vast majority of cases, by the expert panel with or 
without the agency’s staff in attendance (table 52). The site visit is conducted by the 
agency’s staff alone in only three out of 45 cases. 

Six respondents selected the “other” category: in two cases agency’s staff members 
usually attend, but their presence is not formally required or defined; in one case 


agency’s staff members are not always present in all site visits panels; and in another 
case the agency’s staff is in attendance only for part of the visit. 

Table 52. 


WHO CONDUCTS THE SITE VISITS? 

RESPONSE 

COUNT 

RESPONSE % 

The expert panel with a member of the Agency’s staff in attendance 

32 

71.1 

The external expert panel alone 

12 

26.7 

Other (please specify) 

6 

13.3 

The Agency’s staff alone 

3 

6.7 

answered question 

45 

skipped question 

6 


The respondents’ average duration of a site visit (table 53) varies from 1.7 to 2.9 days, 
depending on the type of quality procedure in question. 

Table 53 


WHAT IS THE AVERAGE DURATION OF THE SITE 
VISITS IN DAYS? 

RESPONSE 

AVERAGE 

RESPONSE 

TOTAL 

RESPONSE 

COUNT 

per institutional-level external quality procedure? 

2.9 

101 

35 

per programme-related external quality procedure? 

1.9 

60 

32 

per thematic external quality procedure? 

1.7 

17 

10 

per subject-related external quality procedure? 

2.1 

23 

11 

answered question 

45 

skipped question 

6 


Interviews are the most commonly included element in site visits, followed by a tour 
of the facilities, examination of documentary evidence, and a final meeting with 
the management to present the main findings and conclusions (table 54). Classroom 
observation, as a way of assessing and reporting on the overall quality of learning and 
teaching at the institutional level, ranked lowest. Research outcomes, publications, 
facilities and equipment were reported under ‘other’. There was little difference 
between the 2003 survey and 2008 survey findings. 

Table 54. 


WHICH ELEMENTS ARE INCLUDED IN THE SITE VISIT? 

RESPONSE 

COUNT 

RESPONSE % 

Interviews with members of the institution 

44 

97.8 

Tour of the facilities 

42 

93.3 

Examination of the documentary evidence 

42 

93.3 

Final meeting with the management to the present main findings and 
conclusions 

37 

84.4 

Interviews with stakeholders from outside the institutions 

24 

57.8 

Perusal of student’s work (dissertations, etc.) 

23 

51.1 

Classroom observations 

17 

35.6 

Other 

10 

22.2 

answered question 

45 

skipped question 

6 


Management staff was the group most frequently interviewed during the site visit 
(95.6%), followed by teaching staff (93.3%), students (93.3%), administrative staff 
(86.7%) and research staff (73.3%). Graduates ranked lowest (60%) (table 55). Additional 
interviewees were identified under the ‘other’ category, including: self-evaluation group; 
HEI quality assurance staff; employers; collaborative partners (if relevant); support 
staff; regional authorities; stakeholders; and representatives of business/ industry. 


Table 55. 


WHO IS INTERVIEWED AT THE SITE VISIT? 

RESPONSE 

COUNT 

RESPONSE % 

Management staff 

43 

95.6 

Teaching staff 

42 

93.3 

Students 

42 

93.3 

Administrative staff 

39 

86.7 

Research staff 

33 

73.3 

Graduates 

27 

60 

Other (please specify) 

11 

24.4 

Interviews are not used 

1 

2.2 

answered question 

45 

skipped question 

6 


Considering the responses to Question 56 (Who is interviewed at the site visit?), 
Question 50 (Who, typically, is included in the institution’s self-evaluation group?) 
and Question 51 (What sort of statistical data is used in the self-evaluation?) it can be 
deducted that the management and teaching staff followed by students form the major 
source of evidence that the evaluators directly rely on via interview, and indirectly 
count upon via statistical data. 

5.4 Conclusion 

The fact that it is relatively uncommon for institutions to include graduates in the 
self-evaluation panels or to consider labour market statistics as relevant data in such 
evaluations, suggests that there may be scope for making the self-evaluation more 
outward-looking. It would be a concern if graduate opinion and labour market data did 
not form part of an institution’s self-evaluation considering the EHEA focus on learning 
outcomes and the Lisbon Strategy. 

Site visits are widely used by the respondents of this survey. They are usually 
conducted by an expert panel, which sometimes has some of the agency’s staff in 
attendance. The ‘average duration’ of a site visit is 2.9 days for institutional-level 
external quality procedure, 2.1 days for subject-related procedure, 1.9 for programme- 
related procedure and 1.7 for a thematic one. 

In general, the findings presented in Chapter 5 are similar to the 2003 survey results. 
However, with the current shift towards learning outcome -oriented programmes, it 
is reasonable to expect that in the future more attention will be focused on students, 
graduates and employers when assessing the overall quality of an institution. 


Chapter 6: Reporting the outcome 
of the agency’s external quality 
procedure 

6.1 Content of the published report 

Agencies were asked if there is a publicly available report (table 56). Thirty seven 
agencies answered affirmatively, while nine agencies gave a negative response. 

Table 56. 


IS THERE A PUBLICLY AVAILABLE REPORT? 

RESPONSE 

COUNT 

RESPONSE % 

Yes 

37 14 

80.4 

No 

9 

19.6 

Other (please specify) 

8 


answered question 

46 

skipped question 

5 


The respondents who answered “no” gave the following explanations: 

• the reports are not publicly available yet, but will be in the future (answers given 
by four Associates); 

• the circulation of the reports is limited (one Associate and one Candidate 
member); 

• only some parts of the evaluation, e.g. assessment of the teaching activities of the 
academic staff, are published (one Associate); 

• the decisions are public, but the reports are published only if the institution gives 
its consent and, in case of an accreditation, if the decision is positive (two Full 
members). 

The majority of the published reports include conclusions (87%), recommendations 
(78.3%) and analysis (78.3%). Empirical evidence (54.3%) and conditions (41.3%) rank 
lower. Four agencies answered that the report is not published (table 57). 

Table 57. 


WHAT IS THE CONTENT OF THE PUBLISHED REPORT? 

RESPONSE 

COUNT 

RESPONSE % 

Conclusions 

40 

87 

Analysis 

36 

78.3 

Recommendations 

36 

78.3 

Empirical evidence 

25 

54.3 

Conditions 

19 

41.3 

Others (please specify) 

11 

23.9 

The report is not published 

4 

8.7 

answered question 

46 

skipped question 

5 


This is the same number as was reported in Chapter 3.2.6., where 37 agencies responded that they use a published report within 
their external quality procedures. The percentage is different in the two chapters because the number of respondents who 
completed the question differs. 


One agency answered that it does not use the term “conditions” but there are three 
categories of recommendation one of which is deemed “essential” and which would 
be considered a condition. Another mentioned that the contents of public reports 
are: an improvement plan, CV’s of the experts, “declaration of independence and 
confidentiality” undersigned by each expert and a list of some of the documents 
reviewed by the panel, for example publications by the programme and students’ work. 
Two agencies publish only the final decision and two agencies publish summary reports 
of several evaluations. 

6.2 Number of reports published by the agency over the past three calendar 
years (i.e. January 2005-December 2007) 

Table 58. 


NUMBER OF REPORTS PUBLISHED BY YOUR AGENCY 
OVER THE PAST THREE CALENDAR YEARS (I.E JANUARY 
2005-DECEMBER 2007) 

RESPONSE 

AVERAGE 

RESPONSE 

TOTAL 

RESPONSE 

COUNT 

How many institutional evaluation reports has your 
Agency published? 

71 

2282 

32 

How many programme evaluation reports has your 
Agency published? 

217 

6953 

32 

How many subject evaluation reports has your Agency 
published? 

50 

906 

18 

How many thematic evaluation reports has your Agency 
published? 

8 

140 

17 

answered question 

39 

skipped question 

12 


Programme evaluation reports are the most commonly published reports, with a total 
number of 6,953 reports published from 2005 to 2007 (table 58). The biggest number 
of programme evaluation reports is published by NAA Russia (1,787) followed by NVAO 
the Netherlands and Flanders (1,700). Nine agencies have published between 100 and 
600 programme evaluation reports. Nine agencies have published between 11 and 
76 reports. Eleven agencies have published between o and 10 programme evaluation 
reports. 

Institutional evaluation reports are the second most common with 2,282 reports. 
One agency is responsible for the publication of 1,784 reports (NAA Russia). The two 
next biggest publishers of institutional evaluation reports are the QAA, UK and the 
Commission for Accreditation and quality assessment, Serbia. They have published 
131 and 80 reports respectively. 21 agencies have published between one and 48 
institutional evaluation reports over the three years (2005-2007). Eight agencies 
answered that they have not issued institutional evaluation reports. 

The total number of subject evaluation reports and thematic evaluation reports over 
the three years is 906 and 140 respectively. Four agencies have published between 
118 and 300 subject evaluation reports and five agencies between two and 50. Nine 
agencies have not published any subject evaluation report. One agency (QAA, UK) 
has published 93 thematic evaluation reports and another agency (CNVSU, Italy) 29 
reports. Five agencies answered between one and seven thematic evaluation reports 


over the past three years, and to agencies have never issued thematic evaluation 
reports. 

6.3 Consultation before the publication of the report 
Table 59. 


WHO IS CONSULTED BEFORE THE REPORT IS PUBLISHED? 

RESPONSE 

COUNT 

RESPONSE % 

The institution(s) that is(are) the subject of the external quality procedure 

32 

82.1 

Other (please specify) 

9 

23.1 

Central government 

4 

10.3 

Professional organisations 

3 

7.7 

Industry and labour market 

2 

5.1 

National associations of HEIs (e.g. Rectors’ Conference ...) 

2 

5.1 

Regional government 

1 

2.6 

answered question 

39 

skipped question 

12 


In the great majority of cases the institutions that are the subject of the external quality 
procedure are consulted before the report is published (82 %) (table 59). 

The second most consulted body is the central government (10 %), followed by the 
professional organisations (8%), the industry and labour market (5%) and the national 
associations of HEIs (5%). Only one agency answered that the regional government 
is consulted before publishing the report. Two agencies answered in the category 
“other” that agency’s board or commission on accreditation is consulted and one agency 
answered that a federal body of supervision in education and science is consulted in 
addition to the institution before publishing the report. 

6.4 Who publishes the report? 

In most cases, the agency publishes the report (93%). The report is rarely published by 
the central government (5%) and by the evaluated institution (2%). In Germany the 
agencies are required by the German Accreditation Council to publish the reports on all 
degree programmes using a database provided by the HRK. (table 60). 

Table 60. 


WHO PUBLISHES THE REPORT? 

RESPONSE COUNT 

RESPONSE % 

The quality assurance agency 

40 

93 

Others (please specify) 

5 

11.6 

Central government 

2 

4.7 

The evaluated institution 

1 

2.3 

Regional government 

0 

0 

answered question 

39 

skipped question 

12 


Chapter 7: Follow-up procedures and 
meta-evaluation 


When dealing with the follow-up procedures that are used by the QA agencies it is 
important to distinguish current methods from future plans. Another important 
consideration is the existence of a small group of agencies which are currently defining 
or adapting their procedures and methodologies for follow-up. The analysis will focus 
on current methods and will not comment on the merits or otherwise of any future 
plans declared by respondents. 

Of the 45 respondents to the question “who is formally responsible for taking action/ 
following up on recommendations/conclusions of the report?”, 34 out of 45 agencies 
reported that institutions are responsible for follow-up on recommendations included 
in the assessment reports (table 61). A minority (13%) indicated that the central 
government is formally responsible. In two cases, there is no formal follow-up to quality 
procedures. 

The number of agencies reporting to be responsible for the follow-up procedures 
is significant (27 out of 45). Multiple answers were possible for this question. In 19 
cases both the agency and the institution have formal responsibility for follow-up. In 
three cases the agency, the institution and the central government all have formal 
responsibility. In other three cases, there is a combined responsibility involving the 
central government and the institution, and in two a responsibility involving the agency 
and a national body (German Accreditation Council). Thirteen agencies declared that 
they are not formally responsible for follow-up. 

As could be seen in Chapter 6, public reports produced by the agencies include 
recommendations in 36 out of 46 cases. Although reports include conditions or 
recommendations, it is possible that institutions (in eight cases) or QA agencies (in 
six cases) are not formally responsible for the follow-up. The respondents’ possible 
interpretations of the question, and particularly the word “formal”, give pause for 
thought, considering that just a little over a half of the agencies (27 out of 45) indicate 
that they are formally responsible for follow-up actions on recommendations. Indeed, 
the responses to the question in table 62 concerning follow-up actions involving 
the agency indicate that 39 agencies out of 44 are involved in follow up actions. Ten 
agencies indicate that the institutions are not formally responsible for taking action on 
recommendations/conclusions of the report. Here also there is a need to question the 
interpretations given to the term “formal”. 

Table 61. 


WHO IS FORMALLY RESPONSIBLE FOR TAKING ACTION/THE FOLLOWING 
UP ON RECOMMENDATIONS/CONCLUSIONS OF THE REPORT? 

RESPONSE 

COUNT 

RESPONSE % 

The institution that is the subject of the external quality procedure 

34 

75.6 

The quality assurance agency 

27 

60 

Central government 

6 

13.3 

Others 

6 

13.3 

There is no formal follow-up/action taken 

2 

4.4 

Regional government 

0 

0 

answered question 

45 

skipped question 

6 


Considering the kinds of actions that are taken for the purpose of follow-up, 27 agencies 
reported that they request action plans from the institutions and 22 make use of a desk- 
based study (table 62). 16 agencies stated they use a follow-up site-visit and almost the 
same number (17) reported they use a repeat of the external procedure. 

However the key finding here is the large number of agencies that are using more 
than one of the follow-up actions listed in the question, as only 13 of the 44 selected 
only one of the options. 

Table 62. 


WHAT KIND OF ACTIONS ARE TAKEN FOR THE PUPOSE OF 
FOLLOWING UP ON RECOMMENDATIONS/CONLCUSIONS OF THE 
REPORT? 

RESPONSE 

COUNT 

RESPONSE % 

Request for action plan 

27 

61.4 

Desk-based study by quality assurance agency 

22 

50 

Repeat of external quality procedure 

17 

38.6 

Other (please specify) 

17 

38.6 

Follow-up site visit 

16 

36.4 

answered question 

44 

skipped question 

7 


Of the 13 agencies that indicated that they do not have formal responsibility for 
follow-up (table 6r), it is interesting to observe that eight of them indicated, hovewer, 
that they were using two or more follow-up methods. It is common practice among the 
agencies (78%) that do have formal responsibility for follow-up to use two or more of 
the methods listed. 

Of the 17 responses under “other”, four specified that the follow-up procedures were 
under preparation but not in place yet, two explained that the follow-up depends on the 
specific quality assurance procedure used and two said that the follow-up depends on 
the conditions or recommendations of the procedure. 

Twenty five respondents out of 45 (56%) affirmed that they have in place a systematic 
meta-evaluation of the conducted external quality procedures carried out in the 
agencies’ primary domain. From the comments it can be deduced that, out of those 20 
respondents who gave a negative answer, four agencies are new ones, and thereby do 
not have the meta-evaluation process in place as yet. Two respondents indicated that 
the processes are partly in place, but they are not necessarily systematic. 

Twenty two agencies reported that they themselves conduct the systematic meta- 
evaluation processes (table 63). In general the meta-evaluation is conducted by the 
agency, in some cases with the support of external experts. A singular arrangement can 
be found in Germany where the German Accreditation Council organises the meta- 
evaluation process for all the agencies. 

Finally, there is no significant connection between the use of meta-evaluation 
procedures and the way follow-ups are organised and conducted in terms of 
responsibility. 


Table 63. 


WHO CARRIES OUT THE SYSTEMATIC META-EVALUATION OF THE 
CONDUCTED EXTERNAL QUALITY PROCEDURES IN YOUR AGENCY'S 
DOMAIN (PROCESS AND REPORT)? 

RESPONSE 

COUNT 

RESPONSE % 

The quality assurance agency 

22 

78.6 

Others (please specify) 

9 

32.1 

An external agency 

2 

7.1 

Central government 

1 

3.6 

Regional government 

0 

0 

There is no systematic meta-evaluation of the conducted evaluations 

0 

0 

answered question 

28 

skipped question 

23 


Conclusions 

The responsibility of the follow-up process is a matter generally carried by the agencies 
and/or the higher education institutions. A significant number of the agencies declared 
that responsibility is borne both by the agency and by the institution. 

Follow-up methods are diverse and it is common for more than one method to be 
used by the same agency. 

Meta-evaluation is a function that is generally placed under the responsibility of the 
agencies. In some countries there is another authority that undertakes that function. 


Chapter 8: Future developments 

In the final part of the survey, the agencies were asked to answer questions relating to 
future developments of the quality assurance sector in their primary domain. 

The implementation of the ESG, and the establishment (or renewal) of national 
qualification frameworks (NQF) have clearly influenced the agencies’ procedures and 
will continue to do so. 

The spotlight is currently on issues such as the development of quality assurance 
methodology for e-learning and joint degrees, and the importance of confidence and 
mutual recognition between agencies. 

A significant development that occurred following completion of the survey was 
the European Quality Assurance Register for Higher Education (EQAR), which 
was formally established on 4 March 2008, is expected to “serve as an instrument 
to improve the quality of agencies and to promote mutual trust among them”. The 
emergence of the EQAR had been expected and respondents’ opinion was sought 
concerning it. 

8.1 Agencies in the next two years 

As shown in table 64, half of the respondents are expecting, over the next two years 
(i.e. by January 2010), some developments regarding the establishment of National 
Qualifications Frameworks (NQF) and the introduction of different methodologies 
for external quality procedures. More than half of the respondents plan to revise their 
external quality procedures. A total of 13 agencies expect change in legislation and NQF 
against 16 who expect change in external quality procedures. 

Table 64. 


PLEASE DESCRIBE THE EXPECTED DEVELOPMENTS IN THE QA SECTOR 
IN YOUR AGENY'S PRIMARY DOMAIN IN THE NEXT TWO YEARS (I.E. 
BY JANUARY 2010). 

RESPONSE 

COUNT 

RESPONSE % 

Revision of external quality procedures 

27 

57.4 

Establishment of National Qualifications Framework 

23 

48.9 

Introduction of different methodologies for external quality 
procedures 

22 

46.8 

Commencement of new higher education legislation 

15 

31.9 

Reorganisation or merging of quality assurance agencies 

15 

31.9 

Other, please specify 

11 

23.4 

answered question 

47 

skipped question 

4 


Fifteen agencies, throughout various European countries, indicated forthcoming 
reorganisation or merging of QA agencies involving themselves. These included, for 
example: VLHORA Belgium, HETAC Ireland, the three Austrian agencies, ACQUIN 
Germany, Slovenian Council for Higher Education, the Serbian Agency for Science and 
Higher Education, YODEK Turkey and ECA. 


8.2 On external reviews of agencies 

Many respondent agencies (20 out of 46) indicated that they are currently preparing 
for being externally reviewed (table 65). 39% of respondents plan to subject themselves 
to external reviews in the future. 14 respondents (30%) have been externally reviewed 
already and 6.5% are not planning to subject themselves to external review in the 
future. Two of the four agencies who answered “other” are externally audited through 
the ISO standard. 

Table 65. 


PLEASE INDICATE WHICH OF THE FOLLOWING ASSERTIONS APPLY: 

RESPONSE 

COUNT 

RESPONSE % 

My Agency has been externally reviewed 

14 

30.4 

My Agency is currently preparing for being externally reviewed 

20 

43.5 

My Agency plans to subject itself to external reviews in the future 

18 

39.1 

My Agency is not planning to subject itself to external review in the 
future 

3 

6.5 

Other (please specify) 

4 

8.7 

answered question 

46 

skipped question 

5 


Table 66 demonstrates what kind of arrangements the agencies have/will have for the 
coordination of their review. It is interesting to note that five agencies have indicated 
that their past evaluation was coordinated by ENQA, when in reality in January 2008 
there had only been three ENQA-coordinated reviews - they had started only in 2007. 
The same applies to the evaluations that were going on in January 2008: there was just 
one ENQA-coordinated review taking place, instead of three. This confusion might 
be due to the fact that the difference between nationally coordinated and ENQA- 
coordinated reviews is not clear to all agencies. Several of the agencies that indicate 
their future evaluations to be coordinated by ENQA operate in an international 
domain. 

Table 66. 


PLEASE INDICATE HOW THE REVIEW OF YOUR AGENCY HAS BEEN/IS/WILL BE 
COORDINATED. 

RESPONSE 

COUNT 


PAST 

EVALUATIONS 

CURRENT 

EVALUATIONS 

FUTURE 

EVALUATIONS 

Coordinated by ENQA 

5 

3 

13 

21 

Nationally coordinated 

8 

9 

8 

25 

Other (please specify) 

6 

answered question 

40 

skipped question 

11 


8.3 On the Register 

The majority of respondent agencies (28 out of 46) intend to apply for inclusion in the 
recently established European Quality Assurance Register in Higher Education, EQAR 
(figure 67). 


Figure 67. Is your Agency planning to apply for the Register? 



Out of the eight agencies that do not intend to apply for inclusion five are ENQA full 
members including HAC Hungary, CNVSU Italy and German Accreditation Council 
(the latter because of its role as a meta-evaluator); two associates (from Iceland and 
Israel) and one affiliate (ECA). Six of the 10 agencies who answered “other” stated that 
they had not officially decided yet whether to apply or not; two (from Latvia and Slovak 
Republic) indicated that they would apply if they can afford to and another two would 
apply if, in their estimation, inclusion is either necessary or beneficial to the agency. It 
is interesting to note that ten agencies who plan to apply for inclusion in the Register 
also intend to engage ENQA for the coordination of their future evaluation. Only 
six agencies that plan to apply for inclusion will organise their future evaluation at a 
national level. It might be that they see the ENQA-coordinated review as having more 
legitimacy than the national one, though in practise there is not much of difference 
between the two, especially when the national review is conducted according to the 
ENQA guidance. 

The most commonly expected benefits from inclusion in the Register are 
international recognition as a quality assurance agency (85%) and credibility (82%) 
(figure 68). It is interesting that nine agencies out of 36 are expecting national 
recognition as QA agency/accreditation agency after inclusion in the Register. The 
nine agencies include CAQA Serbia, ECCE, FINHEEC, ZEvA, three anonymous, AQU 
Catalonia and ANECA. This might be because the inclusion in the Register will in some 
cases be included in and required by the Education Law. In Finland it has been planned 
that in the future the universities and HEIs could choose any of the agencies from the 
Register to do their quality procedures. 

Figure 68. If your Agency plans to apply to be included on the Register, what are the benefits 
that you expect to follow? 
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The general feeling towards the newly established EQAR is positive as 66% of the 
agencies (31 out of 47 respondents) see it as a useful tool to increase recognition. Twenty 
one out of the 47 respondents think that the Register will promote confidence between 
agencies and 17 consider it as an opportunity. Only one agency sees it as a threat, 
mainly because of the additional bureaucracy, effort and cost what it brings. This is 
feeling is shared by several of the 13 agencies who answered “other”. One respondent 
indicated that “if well managed it could prove a useful and complementary tool to 
ENQA membership, serving as a quick recognition tool as opposed to the support 
and network function offered by ENQA. However, if badly managed it will cause 
duplication of effort and an increase in burden on agencies” (QAA, UK). 


Chapter 9: Findings 

9.1 Introduction 

This final chapter seeks to summarise the key findings from the ENQA Quality 
Procedures Project (QPP) survey of quality assurance (QA) agencies across the 
European Higher Education Area (EHEA), not as a straightforward summary of what 
has been shown in the previous chapters but from the perspective of a number of ‘cross 
cutting themes’. More subjective than the detailed reporting of the earlier chapters, 
selected contents are brought together to provide insights not only into what is being 
done, and why, but what patterns of activities can be identified, and the related 
‘pressures’, expectations and opportunities that QA agencies are subject to. The chapter 
comments on some of the changes since the 2002 survey, and also looks to the future. 

9.2 Key findings 

Across the EHEA new agencies continue to be established [Chapter 1: tables 1 and 
2], whilst the functions, objectives and priorities for agencies are also changing [see: 
sections 3.2 and 3.3]. Yet the core business of providing information on the quality of 
higher education remains, with increasing levels of external (stakeholder) involvement. 
A high level summary of the more factual aspects of key findings is provided in the 
Executive Summary. 

9.3 Cross cutting themes 

9.3.1 QA AGENCIES PROVIDE A STABLE BUT INCREASINGLY IMPORTANT 
ROLE IN EUROPEAN HIGHER EDUCATION. 

The survey responses clearly indicate that whilst most agencies have a national 
‘remit’ this is not always the case. In Spain for example there is an additional regional 
dimension, and in some countries (Austria and Denmark, for example) different parts 
of the HE sector are assured by different agencies. In Germany a ‘market’ in QA has 
been established with a number of ‘competing’ agencies under an accreditation council. 
In addition to the over-arching and integrating role of ENQA with regard to the QA of 
European higher education, one major consortium of agencies with a shared specific 
(methodological) interest has been formed. 

The status of agencies remains formal (most are either recognised through formal 
legal procedures or they are recognised by public bodies) and many aspects of their 
work remain fundamentally the same as in 2002 [see: 3.3.1]. But there is also evidence 
of an increase in the scope and range of the work undertaken by agencies, although 
whether they employ those actually undertaking their evaluations or contract directly 
or indirectly, varies. 

Whilst two-thirds of the agencies are primarily concerned with evaluations at 
programme level, about 40% are primarily or significantly involved in activity at an 
institutional level. Most agencies are not confined to one type of QA procedure. It is 
still the case that within the general portfolio of approaches to QA two general types of 
procedures, evaluations (assessing aspects of quality) and accreditation (making a yes/ 
no judgement on the basis of defined standards and criteria), tend to predominate, with 
audit (looking at strengths and weaknesses of an institution’s QA mechanisms) only 
taking a major role in a fewer number of agencies. 


For many, the frame of reference for external quality procedures remains, as in 
2002, set by the (national/regional) legal context [see: 4.4.1], but the influence and 
impact of the Bologna Process is becoming more marked [see: 9.3.3 and references 
therein]. 

Reporting findings is a key task for all agencies who responded to Quality Procedures 
Survey. Their reports are almost always published (in line with the ESG) and, at least 
for some agencies, in vast numbers. Reporting such volumes is demanding but students 
might reasonably ask for reliable information about the quality of the programmes 
they may be intending to take. Whether such a volume of QA will continue to provide 
a worth wile benefit to cost and thus provide sustainable may be demonstrated by any 
subsequent Quality Procedures Survey. 

Higher education institutions are, in the substantial majority of procedures, 
formally required to address recommendations [2.4.2] and especially those set out in 
any published reports. Often this is a legal requirement but even so, the authority of 
agencies is in part dependent on their ‘credibility’. This can be and is reinforced by 
agencies being subject to external quality assurance themselves on a regular basis (see 
ESG standard 3.3). It is interesting but not surprising to see that all of the agencies 
that responded had, were in the process of, or were planning to undergo an external 
evaluation. 

The survey returns demonstrate that agencies are carrying out a wider range of 
activities with an increasingly ‘public’ profile, but are they resourced to undertake an 
expanding range of activities? Agencies generally seem to agree (if not always ‘strongly’) 
with the statement that they are adequately resourced, although there are just a few 
who ‘disagree strongly’, and in one case record severe under funding / understaffing. 

The source(s) of an agency’s funding raises some interesting points. Funding from 
government(s) is the main source for most agencies, followed by higher education 
institutions, with ‘grants’ providing a (significant) income for some [2.5.3]. Can these 
responses ‘square’ with the ESG requirement for agencies to be ‘independent’? This 
complex but important aspect is considered in detail in Chapter 2 (section 2.7), and 
it is clear that there is no simple answer and, as the ENQA Quality Convergence 
study (2005) showed, ‘although national quality assurance systems converge to some 
extent, the practicalities in national systems may differ without compromising shared 
principles.’ 

9.3.2 AGENCIES IN TRANSITION 

As well as the formation of new agencies mentioned above the survey also revealed 
restructuring of agencies taking place. In France, for example, the Comite National 
d’Evaluation (CNE) has been integrated into the new Agence d’Evaluation de la 
Recherche et de l’Enseignement Superieur (AERES) (which covers all institutions and 
research bodies as well), whilst in Denmark a new law in has changed the role of the 
long established Danish Evaluation Institute (EVA) and created a new accreditation 
body. But it is not just major structural change that is occurring; three quarters of 
agencies responded that they have changed their QA approach recently or that they 
are about to do so in the near future, with most of these changes of a ‘significant’ 
nature. European higher education is in a period of dramatic flux, driven in part by 
the Bologna Process, but also by a range of other national and international pressures 
including ones that are political, pedagogical, demographical and employment- 


related. ‘Quality assurance in HE has undergone a highly dynamic evolution and this 
process is continuing...’ [2.4.1] and this is reflected in changes in the division of QA 
responsibilities between agencies and institutions; subject evaluations are often the 
responsibility of the institutions, whilst programmes may be covered by institutions 
or agencies and quality assurance procedures of institutions are generally the 
responsibility of the agencies. But changing patterns in the overall work of agencies 
appear to be emerging with, in several cases, a shift away from programme evaluation 
either being instigated or discussed. There are exceptions to the rule however. 

Agencies are not just revising procedures but are adding new activities to their 
portfolio of work [2.4.4]. Many respondents to the survey noted that they were 
changing their methods. Not only are these procedural but they are often accompanied 
by changes of a fundamental nature, with moves from those based largely on the use 
of input measures (entry qualifications, hours of work, staff qualifications, library and 
IT facilities, etc) to ones that are, at least in principle, based on ‘learning outcomes’. 
Such changes are already well under way, and supported by a wider range of objective 
criteria (qualifications frameworks, ‘benchmarks’ (subject and Tuning), the ESG, and a 
continuing ‘political rhetoric’ about the importance of learning outcomes. But quality 
assurance focussing on learning outcomes will continue to provide QA agencies and 
those they assure with some challenges. If learning outcomes are really all that matter 
should quality assurance merely look at the validity and valid assessment of these? 

Of course not, but finding the right balance that will both provide quality procedures 
that are flexible enough to adapt to a range of increasingly different, innovative and 
complex learning environments, whilst still providing a (conservative basis for) public 
reassurance will be yet another (significant) challenge. Quality assurance should not 
stifle innovation and development yet it should be clear about the basic purposes that 
society expects of it. 

As pointed out in section 3.3.1: the increasing “involvement of stakeholders (in 
agencies in terms of functions objectives and priorities) is an indicator of higher 
education’s shift from a teaching to a learning focus, its relationship to society, industry 
and commerce, to employers and professional needs”. 

9.3.3 IMPACT OF THE BOLOGNA PROCESS 

For many agencies the frame of reference for their (external) quality procedures 
remains, as in 2002, set by the (national/regional) legal context [4.4.1], but the 
influence of Bologna is becoming more marked. Agencies are changing their QA 
procedures in significant ways, with features of the Bologna Process being amongst the 
drivers. Of these, alignment with the ESG was shown in the survey to be particularly 
important [2.4.4]. Qualifications frameworks are also becoming increasingly important 
reference points for QA procedures - with agencies often involved the development of 
their national qualifications frameworks and in time, no doubt, the self certification 
of these against the FQEHEA. Other specific criteria for agencies and their quality 
procedures were mentioned individually and there were a few who include the OECD/ 
UNESCO guidelines on quality provision in cross border higher education. But perhaps 
the most significant change from the 2003 survey is that, whilst then the notion of 
specific standards and criteria was ‘emerging as a common feature’, a number of these 
have not only ‘emerged’ but are in widespread use. It is perhaps interesting to note that 
standards and criteria initiated within the HE sector and developed by widespread 


consensus (the FQEHEA and the ESG) are referred to far more frequently than, for 
example, Tuning and various ‘quality labels’. 

One of the more paradoxical findings from the survey is that whilst responding 
agencies are, or are hoping to be, members of ENQA, and there is a general expectation 
within the ESG that agencies will carry out their quality procedures through a 
four stage model, there are a surprisingly high number of returns that indicate that 
procedures do not follow such a model. As pointed out in 3.3.2 there are several possible 
reasons for this, and various inferences as well, and most agencies do follow the four- 
stage model. With a likely increase in mobility of students and trans-national education 
in general this could be an interesting point to focus on in any following survey. 

9.3.4 WORKING TOGETHER 

‘External QA agencies operate in a socially and technically complex domain where 
there can be tensions between generally accepted principles. With governments, higher 
education institutions and quality assurance agencies all on the same playing field there 
can be uncertainties about who is in charge of what’ [2.5.1]. 

The survey clearly indicates a division of QA ‘responsibilities’ - subject and 
programme approval and evaluation by institutions, programme and institutional 
approval and evaluation by agencies, institutional approval generally by (or on the 
recommendation of) agencies. But the responses also indicate changing patterns 
emerging in this balance of work between institutions and agencies, with shifts away 
from external programme evaluation either having taken or taking place, or being 
discussed; although there are exceptions. 

Site visits are where ‘working together’ really should ‘come together’, and Chapter 
5 provides a great deal of specific information on this. It identifies how institutions’ 
self evaluation groups are formed and work, who is represented on them, and what 
data sources support their reports. Details of the range of practices in the site visits 
themselves are also provided, including the duration, the elements included and 
the types of interviewees. The addition, on average, of only one extra day for an 
institutional evaluation over a programme evaluation might be a surprise to some, but 
the detailed scrutiny of primary evidence in the latter and a focus on v in the former 
may explain this, in part. In comparison to 2003 there is a reduction in the extent to 
which agency staff is directly involved in all of the site visit activities, but this is offset 
by increasing participation of students in the panels. The continued limited use of 
graduate surveys and labour market information is perhaps a surprise in light of the 
current (political) pressures and emphases on ‘employability’, but these may not have 
yet worked through fully into the detail of the internal institutional or external quality 
procedures (something for the next survey to look at and compare?). 

The relationships between ‘stakeholders’ and QA agencies cover a variety of aspects, 
including funding, the selection of QA procedures and their criteria, and the making 
and dissemination of decisions / judgements. Funding is discussed briefly above [and 
more extensively in 2.5.3]. I n terms of ‘responsibility for agencies’ operations’, more 
than three quarters of the respondents stated that external stakeholders do not have 
any responsibilities at all. Amongst those agencies that did acknowledge stakeholder 
responsibilities these included membership, at various levels, of committees and panels 
and consultation(s) when establishing or changing QA procedures and criteria. It is 


important to note that ‘stakeholder involvement should not be mistaken for influence of 
external stakeholders on the agencies’ operations.’ [2.7] 

Agencies draw on a wide range of experts when assembling the panels that will 
undertake their quality procedures, using independent national and international 
experts to assist in making peer judgements, with students being used increasingly (c.f. 
2003) but agency staff playing a smaller role in the panels themselves [see 4.2.2]. There 
are slight differences in the ways in which agencies work with their experts; in terms 
of functions, it is usually the agencies that determine the choice of QA procedure, 
the preparation of any guidance materials, and the practicalities of any site visits. By 
contrast, experts in many cases take an increasingly important responsibility and ‘load’ 
when it comes to the preparation of any reports [4.3.2] although any formal decisions 
and report publication revert in most cases to the agencies. 

The quality assurance of collaborative provision was not looked at in the 2003 survey 
so there is no baseline for the results presented in section 4.5. Trans-national HE is 
increasing rapidly and is predicted to become more important, particularly within 
the promoted European and globalised agendas around student and labour mobility. 
There are ‘hints’ in the survey data of an increasing awareness of the need for closer 
collaboration in trans-national and trans-regional QA to meet expectations, particularly 
of students and their employers, in shared standards and criteria that can underpin 
‘fairness’ and ‘comparability’. 

Chapter 7 deals, more substantially, with follow-up procedures and meta-evaluations. 
There are a variety of (context dependent) approaches and actions but the general 
impression is that where matters are raised by external evaluations that do need 
to be addressed there are ways and means of ensuring that changes occur within 
institutions. One area the survey did not address in detail was the extent to which the 
information in the numerous reports published by agencies is, or could be, collated 
to provide more generic information on and assistance to improving practice, both 
of the quality assurance mechanisms within institutions generally (as opposed to the 
specific recommendations that are/have to be acted on), and also for external quality 
procedures more generally. 

9.4 Future directions 

Future developments are addressed specifically in Chapter 8. Agencies have been, 
currently are, and expect to be centres where much change is happening. These 
changes are not just minor revisions to quality procedures, although the survey shows 
that these are taking place on an almost continuous basis. There are much more 
substantial changes occurring as well. Some of these, and their implications, can 
provide an insight into general ‘directions of travel’ that agencies may choose, with 
more or less ‘encouragement’. 

Agencies, governments and stakeholders may have somewhat different views on 
quality and particularly on the most effective ways to assure and improve/enhance it. 
The tolerance for diversity in the approaches to quality assurance can be influenced by 
context and perceptions. These can be significant, for example, in the cases where the 
requests of students and employers and even tax payers have resulted in the (additional) 
accreditation of programmes [2.4.1]. 

The recent and increasing emphasis in many European countries on ‘employability’ 
and ‘graduate skills’ is perhaps under-reflected in the priorities recorded in many of the 


quality procedures reported here (there is an inevitable time lag between policy and 
implementation), but if this trend continues it may be likely that greater emphasis on 
these topics will be seen in any future analyses of quality procedures. 

With increasing mobility of students and employees and growth in trans-national 
higher education, it is likely than any future survey will also find greater emphasis on 
the quality assurance of collaborative provision. Whether this is through institutions 
expanding their provision beyond their own national/regional boundaries as ‘sole 
traders’, or through ‘joint ventures’ or ‘franchising’, students surely have a right to the 
same level of quality of provision and standards of awards wherever and however they 
study. The quality assurance of collaborative provision is currently a relatively minor 
interest [see: 4.5]; this may well not be the case in a few years time. It remains to be 
seen whether new approaches to quality assurance of joint/trans-national provision 
are more likely to be through joint QA or through some form of ‘mutual recognition’. 

It is however interesting to note that one of the respondents to the Quality Procedures 
survey was not a single agency but a consortium of agencies from various different 
national and regional jurisdictions, seeking to work together in what is, in practice, 
difficult legal territory. 

Chapter 3 includes data that shows an interesting development in terms of 
international activity: although direct work on international comparisons is not 
currently amongst agencies’ top objectives (as in 2003) there are now far more agencies 
devoting staff time and effort to international work. There seems to be an underlying 
message that, whilst agencies certainly need to keep abreast of what is happening on an 
international basis in order, for example, to benchmark themselves to retain credibility, 
there may also be an element of planning for a broader scope of activity in future. 

9.5 Final conclusions 

The 2008 Quality Procedures survey has provided a wealth of data on what agencies 
have been doing, are doing and are planning to do. It is clear that there are not only 
an increasing number of agencies but that, as well as maintaining much of their more 
‘traditional’ portfolio of work, many agencies are (being asked/expected) to change and 
broaden their remit and the scope and types of their activities. This is, perhaps, not 
surprising; with more and more students from a greater diversity of social backgrounds 
going into an increasingly diverse range of higher education provision it is right that 
there is an increasing demand for more detailed information about the quality of higher 
education. 

Students are increasingly involved in the procedures themselves, and demanding 
new sorts of information; employers are faced with an ever-widening range of graduates 
and, if they are to recruit the best for their purposes, they will need to have a better 
understanding of just what particular graduates have learnt and can do. The emphasis 
on learning outcomes will increase. 

But can quality procedures cope with the amount of work required to provide 
reliable information on these at a detailed level? Probably not if the emphasis on 
data collection is merely for accountability and reporting; but if quality assurance 
has quality improvement/enhancement as an agreed and truly integrated focus then 
it is more likely to be able to. It is both interesting and encouraging to note that the 
survey shows, in several places and in different ways, such a change in emphasis; from 
assurance to enhancement could have been another cross cutting theme. Agencies and 


the institutions they are responsible for evaluating are at different stages in the journey 
of moving to explicit external quality assurance. Some have been doing it for decades, 
others only for months. 

It is unrealistic to expect that the same model(s) of quality procedures are 
applicable to all; legal, social, pedagogical and other contexts are different and quality 
assurance must reflect these. Similarly, it is unrealistic to expect that all agencies 
and institutions will instantly have the same levels of understanding and expertise in 
quality procedures. It is clear that the Bologna Process is having a major impact on the 
way that agencies have developed since the 2003 survey; it will be interesting to see 
what further changes occur over the next few years, especially with the introduction 
of the European Quality Assurance Register for Higher Education. Whilst this will, 
undoubtedly, have an impact at the outset, the extent to which it is sustained will, as 
with all such mechanisms, finally be reliant on evidence of its worth. 
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Annex: The Quality Procedures Questionnaire 


Second ENQA survey on Quality Procedures in European Higher Education 

1. Introduction 


The object of this web- based survey is to collect information from ENQA members, candidates, affiliates and 
associates for the purpose of updating the findings of the ENQA report Quality Procedures in European Higher 
Education (ENQA Occasional Paper n. 5, year 2003.) This project is funded by the European Commission through 

its Socrates Programme. 

The survey contains about seventy questions and will probably take at least 90 minutes to complete. The survey 
allows one response per computer (identified by IP address.) You do not have to complete the survey in one 
sitting. You or any other person with access to your computer (the one you used to start the survey) may use 
the Web link provided to go back to previous pages in the survey and update existing responses until the survey 
is completed or the closing date. When using the Web link to re-enter the survey you might need to click the 
refresh button on your web browser to reveal the most recent data. 

Please attempt to answer all of the questions. Many questions make provision for the entry of succinct 
comments. Typically such comments are limited to approximately 100 words. If you exceed the limit you will need 
to shorten your response to progress with the survey. A very small number of questions are marked with an 
asterisk, these need to be answered to progress with the survey. Certain other consistency checks are made by 
the software and if additional input is required before progressing with the survey an appropriate message will be 
displayed in red beside the relevant question. If you click next without making the required input the page will not 
advance;(sometimes you may need to scroll back to see the message). 

The closing date for the survey is 31 January 2008. 

The survey is being conducted an ENQA project group (PG) and steering group (SG). The members of the groups 
are: 

• Nick Harris, (QAA) (SG) 

• Achim Hopbach, (SG), ENQA Board member 
. Peter Cullen, (HETAC), Chair (SG and PG) 

• Fiona Crazier, (QAA) 

• Josep Grifoll, (AQU) Catalonia 
. Helka Kekalainen, (FINHEEC) 

• Tanel Sits HEQAC, (student representative) 

• Kurt Sohm, (Austrian FHR), ENQA Board member 

• Bozana Knezevic, Croatia (HEI representative) 

• Emmi Helle (ENQA), Secretary (SG and PG) 

• Nathalie Costes (ENQA), Alternate Secretary 

The SurveyMonkey Tool is being used to collect the responses. 

The responses will be analysed by the Project Group, and presented in the Second Report on Quality Procedures 
in European Higher Education, to be published by August 2008. 

Your responses will not be used in any ENQA coordinated review of your agency or for any purpose other than 
that stated above. 

If you have any questions about the survey please contact Nathalie Costes Nathalie. Costes@enqa.eu. 

Note on multiple choice questions: The survey contains two types of questions that offer multiple options. The 
more common type (indicated by a square check-box) allows a respondent to select combinations of more than 
one options. The other type (indicated by a circular check-box) requires a respondent to select a single choice. 

In both cases the order of the choices is randomly assigned for each response except in a few cases where a 
particular order is used to enhance clarity. 

Note on length of responses: Where a question includes a space for comments you may enter up to a maximum 
of 600 characters including spaces (approximately 100 words). Responses should be in the English language. 
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Second ENQA survey on Quality Procedures in European Higher Education 


1. Name of your Agency. 


* 2. What is your Agency's relation to ENQA 

o Associate 
o Candidate member 
o Full member 

Affiliate 

Other (please specify) 

3. Name of respondent and contact information 

Name: | | 

E-mail | | 

Address 1: | | 

Address 2: | | 

City/Town: | ] 

State/ Province: | | 

ZIP/Postal | | 

Code: 

Country: | | 

* 4. Several of the questions will ask you to express a view concerning some issue in 
your Agency's primary domain. For many this will be the country in which the 
agency is established but it could be more or less extensive than this. 'Primary 
domain' means the principal territories within which your agency was established 
to operate as an external quality assurance agency. Which of the following best 
describes your Agency's primary domain? 

o International 

A part of the territory of a sovereign state 
o A specific group of sovereign states 

o A country (meaning the whole territory of a sovereign state) 

Please specify 
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Second ENQA survey on Quality Procedures in European Higher Education 


5. Remit or scope of your Agency (Please indicate all that apply here. Also note 
that throughout the survey a set (row, column or matrix) of square tick-boxes 
indicates that you can tick as many boxes as you wish. Circular tick-buttons, on the 
other hand, require you to make an exclusive choice.) 

| | Universities 

I | Higher Education Institutions such as Universities of Applied Science; University Colleges; etc.. 

| | A specific discipline or field of learning 

| | Other (please specify) 

I I 

6. Grant/ refusal of permission to ENQA to publish your responses: 

I give permission for my Agency to be identified with the responses I give in the final report. 

I do not give permission for my Agency to be identified with the responses I give in the final report (your responses will be 
reported anonymously). 


2. Explanatory note 


An external quality procedure[l] could be one of the following: 

• An evaluation of a subject[2], which focuses on the quality of one specific subject, typically in all the 
programmes in which this subject is taught. 

• An evaluation of a programme, which focuses on the activities within a study programme, which in this 
context is defined as studies leading to a formal degree. 

• An evaluation of an institution, which examines the quality of all activities within an institution, i.e. 
organisation, financial matters, management, facilities, teaching and research. 

• An evaluation of a theme which examines the quality or practice of a specific theme within education e.g. 
ICT or student counselling. 

• An audit, which is an evaluation of the strengths and weaknesses of the quality mechanisms established by 
an institution itself to continuously monitor and improve the activities and services of either a subject, a 
programme, the whole institution or a theme. 

• An accreditation process, which builds on the same methodological elements as the other types of 
evaluation, but differs from the other procedures in that judgement is provided according to predefined 
standards to decide whether a given subject, programme, institution or theme meets the necessary level. 

• Benchmarking, which is a comparison of results between subjects, programmes, institutions or themes 
leading to an exchange of experiences of effective practice. 

• where Criteria are seen as checkpoints and benchmarks for assessing the quality of the input and the 
process. 

• where Standards are seen as the expected outcomes of the educational training. For example standards 
defined by professional organisation or legislation. It concerns the competencies that are expected from the 
graduates. 

[1] In this survey the term 'external quality procedure' also covers the terms 'assessment', 'review', audit, etc. 

[2] A subject is for example the subject 'chemistry' within the study programme of medicine. 


3. Background information about the higher education system in your 
primary do... 
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7. Who is responsible for the approval of new subjects, programmes and 
institutions? 



Subjects 

Programmes 

Institutions 

The institutions (self- 
regulation) 

□ 

□ 

□ 

Other 

□ 

□ 

□ 

Central government 

□ 

□ 

□ 

Regional government 

□ 

□ 

□ 

The relevant quality 
assurance agency 

□ 

□ 

□ 


Please specify (other) 

1 

8. Please enter the number of higher education institutions in your Agency's 
primary domain in each of the following categories: 

Private universities | | 

Public universities | | 

Private HEIs such as university colleges, applied universities etc. | | 

Public HEIs such as university colleges, applied universities etc. | | 

9. Please paste a webpage link to an English language fact sheet (if available) 
describing the higher education system in your Agency's primary domain and 
providing a diagram (if available) of the qualification and institutional structures. 
Alternatively please send this by e-mail to: Nathalie. Costes@enqa.eu. This 
incidental information will not be analysed as part of the survey. 


10. Are institutions in your Agency's primary domain subject to compulsory 
external quality procedures by your Agency and/or others? 

o No none are subject. 

0 Yes all are subject. 

Comment 

1 1 


4. About your Agency 
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11. Indicate the extent to which your Agency is recognised as having formal 
responsibility for external quality assurance. 

Not yet formally recognised as such 
o Not fully recognised in the whole of its primary domain 

o Recognised by the competent authorities established for that purpose (or their agents) in the whole of its primary domain 
Recognised explicitly in legislation in the whole of its primary domain 
Please explain 

1 

12. Does your Agency have a legal basis? 

O Yes 
O No 

Please explain 

1 

13. Is your Agency the only competent/responsible body for the external quality 
assurance of higher education institutions based in its primary domain? 

O No 
O Yes 

Comment 

1 

14. Does your Agency undertake external quality assurance activities (at 
institutional or programme level) on a regular basis? 

O No 
0 Yes 
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15. How frequently are the following types of external quality procedures carried 
out by your Agency? (If your Agency does not carry out a particular procedure 
listed select never in the first column of the relevant row.) 


Frequency 


Activity cycle in years 


Consequences: change in 


Is there a formal 
requirement for the HEI to 
take action to address 
recommendations 


of combined 
approaches 
(please 
specify) 

Please specify if applicable 



1 
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16. Has your Agency significantly changed its approach to evaluation in the past 
five years or is it planning to change its approach? 

O No 

o Yes: please specify how your Agency changed (or will change) its approaches and why. 

I I 


17. Indicate your agreement or otherwise with the following assertions. 



Strongly 

disagree 

Disagree 

Agree more 
or less 

Agree 

Strongly 

agree 

In my opinion my Agency has adequate and proportional human 
resources to enable it to organise and run its external quality 
procedures in an effective and efficient manner 

o 

o 

o 

o 

o 

In my opinion my Agency has adequate and proportional 
financial resources to enable it to organise and run its external 
quality procedures in an effective and efficient manner 

o 

o 

o 

o 

o 

In my opinion my Agency has adequate and proportional human 
resources for the continual evaluation and development of its 
processes and procedures 

o 

o 

o 

o 

o 

In my opinion my Agency has adequate and proportional 

o 

o 

o 

o 

o 


financial resources for the continual evaluation and development ' 

of its processes and procedures 

Comment 

1 

18. How many full-time equivalent staff members are (directly) employed by (or 
seconded to) your Agency? 


Number of management staff 

Number of administrative staff 

Number of external quality procedures officers 

Number of research/development officers 

Number of trainees/interns 

Other (please specify in answer to next question) 

Number of staff covering functions that are unrelated to external I 
quality procedures 

19. Please specify the 'other' category selected in the last question. 


20. Indicate the approximate average value of the percentage of funding received 
by your Agency from the following sources. (Please check that the sum of your 
entries is 100.) 

Government 

HEIs (fixed subscriptions only) 

Fees (for specific activities) 

Grants (other than government) 

Other (please specify in answer to next question) 
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21. Please specify the 'other' category selected in the last question. 


22. What kind of general information about itself does your Agency publish for the 
benefit of stakeholders? 

□ Statement of general policy and management plan 

□ Statement of the goals and objectives of the Agency 

□ Statement explaining the division of responsibility for quality procedures between the Agency and higher education 
institutions 

□ Statement setting out the systematic approach to external quality assurance 

□ Other (please specify) 

I ~H 

23. In your opinion, what influence, if any, do external stakeholders have on the 
conclusions and recommendations made in your Agency's reports? 

| | External stakeholders may influence neither the conclusions nor the recommendations in reports 

□ External stakeholders may sometimes influence the conclusions in reports 

□ External stakeholders may sometimes influence the recommendations in reports 
Comment 

1 

24. In your opinion, what share, if any, do external stakeholders have in 
responsibility for your Agency's operations? 

o No share, my Agency has autonomous responsiblity 
Other (please specify) 

i 


5. Typical methodological approach to external quality assurance 


In this part you (the respondent) should focus on your Agency's general approach to external quality assurance 
and on the framework within which this is executed. 
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25. Which of the following stakeholders have a formal role in the specification of 
processes and criteria for the external quality assurance of higher education 
institutions in your primary domain? 

□ Student representatives 

□ International association of HEIs (such as EUA) 

□ Higher education institutions 

□ Government (central/regional) 

□ Industry and labour market representatives (union, employers, etc.) 

□ Professional organisations 

□ National association of HEIs (such as rectors' conference) 

□ Quality assurance agencies 

□ Other (please specify) 

I ~~3 

26. Which body makes the final decision in the specification of processes and 
criteria for the external quality assurance of higher education institutions in your 
primary domain? 

□ The relevant quality assurance agency 

□ Government (central/regional) 

□ Other (please specify) 

I ~1 

27. Does your Agency have a board or council (i.e. a formally established 
governing committee)? 

O Yes 
O No 

Other (please specify) 

If you answered ’No’ to the previous question please skip forward to Q(5) on this page. 
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28. Which of the following stakeholder groups is represented on the Board / 
Council of your Agency? 

□ International association of HEIs (such as EUA) 

□ Professional organisations 

□ Higher education institutions 

□ Government (central/regional) 

□ National quality assurance community 

□ International quality assurance community 

□ Industry and labour market representatives (union, employers, etc.) 

□ National association of HEIs (such as rectors' conference) 

□ Students 

□ Other (please specify) 

I ~3 

29. Who formally appoints the members of the external expert panels used by 
your Agency? 

□ The institution that is the subject of the external quality procedure 

□ Government (central/regional) 

□ Your Agency 

□ Other (please specify) 

~3 
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30. Who may suggest/nominate potential members of the external expert panels 
used by your Agency? 

□ International association of HEIs (such as EUA) 

□ National association of HEIs (such as rectors' conference) 

□ Higher education institutions 

□ The institution that is the subject of the external quality procedure involving the panel 

□ Students 

□ Professional organisations 

□ Industry and labour market representatives (union, employers, etc.) 

□ Your Agency 

□ Government (central/regional) 

□ Other (please specify) 

I ~1 

31. Who makes the final decision on the outcome of your Agency's external quality 
procedures? 

□ Your Agency 

□ The external expert panel 

□ The institution that is the subject of the external quality procedure 

□ Government (central/regional) 

□ Other (please specify) 

I i 

32. Does your Agency always publish relevant processes, criteria and procedures 
prior to execution of external quality procedures? 

O Yes 
O No 
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33. Which of the following processes does your Agency use/require within its 
external quality procedures? 

□ A self-assessment procedure by the institution that is the subject of the external quality procedure 

□ An external assessment by a group of experts irrespective of whether or not there is a site visit 

□ A published report 

□ A follow-up procedure undertaken by the institution that is the subject of the external quality procedure in the light of any 
recommendations contained in the report 

□ Other (please specify) 

~3 

34. Has your Agency established procedures for its own accountability? 

O Yes 
O No 

Comment 

1 


35. In your opinion which of the following accountability procedures are in place 
for your Agency and how effectively is each functioning? 



Not 

Partly 

Established 

and 

Established 

Established 
and very 
effective 


established 

established 

moderately 

effective 

and effective 

An enforced no-conflict-of-interest mechanism in the work of 

o 

o 

o 

o 

o 

your Agency's external experts is established and enforced 

Reliable mechanisms that ensure the quality of any activities 
and material produced by subcontractors, if some or all of the 
elements in your Agency's quality assurance procedure are 
subcontracted to other parties 

o 

o 

o 

o 

o 

An appeals system 

o 

o 

o 

o 

o 

Internal quality assurance procedures which include an internal 
feedback mechanism and an internal reflection mechanism 

o 

o 

o 

o 

o 

Mechanisms for collection and analysis of, and reflection on, 
feedback from experts and reviewed institutions for the purpose 
of informing and supporting your Agency's improvement and 
development 

o 

o 

o 

o 

o 

Provision for periodic independent evaluation and reporting of 
your Agency's conformity with the membership criteria of ENQA. 

o 

o 

o 

o 

o 

A published policy for the assurance of your Agency's own 
quality, made available on its website 

o 

o 

o 

o 

o 

Processes and results which reflect your Agency's mission and 
goals of quality assurance 

o 

o 

o 

o 

o 

Mandatory cyclical external review of your Agency's activities at 
least once every five years 

o 

o 

o 

o 

o 


Other (please specify) 

I 
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36. Indicate the relative importance to your Agency's mission of the following 
where they are functions of your Agency. 



Unimportant 

Of little Moderately 
importance important 

Important 

Very 

important 

Not a 
function 
of my 
Agency 

Recognition and licensing of higher education institutions 

o 

o 

o 

o 

o 

o 

Deciding on the funding of higher education 

o 

o 

o 

o 

o 

o 

External quality assurance of programmes 

o 

o 

o 

o 

o 

o 

Quality enhancement/improvement 

o 

o 

o 

o 

o 

o 

Collecting/disseminating information on quality of HE... 

o 

o 

o 

o 

o 

o 

Recognition of national diplomas 

o 

o 

o 

o 

o 

o 

Development and maintenance of discipline standards 

o 

o 

o 

o 

o 

o 

Quality assurance 

o 

o 

o 

o 

o 

o 

Development and maintenance of the qualification framework 

o 

o 

o 

o 

o 

o 

External quality assurance of institutions 

o 

o 

o 

o 

o 

o 


Other (please specify) 

1 

37. Indicate the relative importance to your Agency's mission of the following 
where they are objectives of the main type of external quality procedures 
undertaken by your Agency. 

Of little Moderately Very Not an 

Unimportant Important 

importance important important objective 


Student (or prospective student) protection 

o 

o 

o 

o 

o 

o 

Quality improvement/enhancement 

o 

o 

o 

o 

o 

o 

Ranking 

o 

o 

o 

o 

o 

o 

Accreditation 

o 

o 

o 

o 

o 

o 

Estimation of graduate learning outcomes 

o 

o 

o 

o 

o 

o 

Accountability 

o 

o 

o 

o 

o 

o 

Transparency 

o 

o 

o 

o 

o 

o 

Make international comparisons with similar 
institutions/programmes 

o 

o 

o 

o 

o 

o 

Make national comparisons with similar institutions/programmes 

o 

o 

o 

o 

o 

o 


Other (please specify) 

I 
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38. Indicate, using the drop-down menus, the importance of the following where 
they are assessed by your Agency's external quality procedures at programme 
level and at institutional level. (Research programmes leading to PhD etc. if 
provided are also relevant here.) 

Programme level Institutional level 


Curriculum/syllabus 

1 

V 

d 

dH 

Employability of graduates 

! 

▼ 

d 

■ 1 

Research strategy 

1 

▼ 

d 

Id 

Collaboration with other higher education institutions 

1 

T 

d 

id 

Study structure 

1 

"W 1 

d 

id 

Cooperation with other subjects, programmes 

1 


d 

d 

Academic and personal support for students 

1 


d 

d 

Learning environments 

1 

T 

d 

III 

Appropriateness of the learning outcomes attained by 

d 

T 

d 

— HI 

graduates 





Supervision of research students 

1 

T 

d 

III 

Teaching and learning methods 

1 

▼ 1 

d 

U 

Research environment 

1 

T 

d 

■i 

Practical training periods (including placements in industry) 

1 

▼ 

d 

■i 

Management, organisation 

1 

▼ 

d 

> 

Research collaboration and links 

1 

T 

d 

Id 

Feedback from students 

1 

T 1 

d 

dfei 

Professional and pedagogical qualifications of staff 

1 


d 

d 

Mission/goals 

1 

T 

d 

III 

Assessment of students including feedback to students 

d 

T 

d 

111 

Internationalisation 

1 

T 

d 

111 

Connection between teaching and research 

1 

T 

d 

Id 

Student retention and completion rates 

1 

T 

d 

id 

Facilities and resources 

1 

▼ 

d 

Id 

Research output of staff and research students 

1 


d 

_tl 

Internal quality assurance procedures 

1 


d 

W 


Other (please specify) 

1 


6. Division of responsibilities and methodological framework for 
procedures in... 
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39. Who are the members of the external expert panel? 

□ Staff members of the quality assurance agency 

□ Professional organisations 

□ National experts representing institutions 

□ Students 

□ National experts representing area of focus 

□ Graduates 

□ Professional practitioners 

□ Employers 

□ International experts 

□ Other (please specify) 

I i 

40. Does your Agency provide training/briefing for the external expert panel? 

O Yes 
O No 

If you answered 'No' to the previous question please skip forward to Q(6) on this page. 

41. Typical duration of any briefing/training: 

What is the duration | | 

(expressed in hours) of 
the training/briefing 
provided to experts? 

42. Please outline the content of any briefing/training: 

1 

43. Characteristics and circumstances of briefing/training provided to panel 
members. 

□ Training/briefing is compulsory for panel chairs 

| | Training is required before a person may be nominated to a panel 

□ Training/briefing is compulsory for all panel members 

□ Training/briefing provided is matched to the experience of each panel member 

□ Training outcomes are assessed by my Agency 

□ Training is provided using distance learning 

□ Other (please specify) 

I I 
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44. Who performs the following functions in the external quality procedures? (If 
the functions are performed in cooperation between the external expert panel 
and the quality assurance agency, please mark both.) 



The external expert panel 

The quality assurance agency 

Choice of basic methodology applied 

□ 

n 

Preparation of the guidelines for the self-evaluation 

□ 

□ 

Preparation of the external quality procedure concept 

□ 

□ 

Contact with the institution 

□ 

□ 

Planning of the site visit 

□ 

□ 

Preparation of the guidelines for the site-visit 

□ 

□ 

Writing of the report 

□ 

□ 

Other 

□ 

□ 

Other (please specify) 




a 


45. Who is responsible for the different parts of the external quality procedure? 
(If both the external expert panel and the quality assurance agency are 

responsible, please mark both.) 

The external expert panel 

The quality assurance agency 

The description and analysis 

□ 

□ 

The conclusions 

□ 

□ 

The recommendations 

□ 

□ 

The formal decision (evaluation outcome) 

□ 

□ 

The report prepared for publication 

□ 

□ 

Other 

□ 

□ 

Other (please specify) 




a 


46. What defines the reference frame for the external quality procedure? 

□ Subject benchmarks 

□ Standards defined by professional organisations 

□ Guidelines of good practice 

□ National Framework of/for Qualifications 

□ A Framework for Qualifications of the EHEA 

□ The stated goals of the institution that is the subject of the external quality procedure 

□ Legal regulation 

□ Other (please specify) 

I I 
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47. What specific criteria and standards are used for your Agency's external 
quality procedures? 

□ The Agency's own published criteria and standards 

□ European Standards and Guidelines 

□ OECD/UNESCO Guidelines on Quality Provision in Cross-border Higher Education 

□ National criteria and standards 

□ Others (please specify all and provide url links) 

~H 

48. Does your Agency have quality procedures for jointly/collaboratively provided 
programmes that are delivered by a group of institutions within one country? 

O Yes 
O No 

Comments 

i 

49. Does your Agency have quality procedures for jointly/collaboratively provided 
programmes that are delivered across different 'jurisdictions'? 

O Yes 
O No 

Comments 

I I 
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52. Do any of your Agency's procedures involve a site visit? 

O Yes 
O No 

Other (please specify) 

ij 

If you answered 'No' to this question please skip forward to the next page. 

53. Who conducts the site visit? 

| | The Agency's staff alone 

| | The expert panel with a member of the Agency's staff in attendance 

| | The external expert panel alone 

| | Other (please specify) 

I ~~1 

54. What is the average duration of the site visit in days 

per programme-related external quality procedure? | | 

per thematic external quality procedure? | | 

per subject-related external quality procedure? | | 

per institutional-level external quality procedure? | | 
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55. Which elements are included in the site visit? 

□ Interviews with stakeholders from outside the institution 

□ Classroom observations 

□ Final meeting with the management to the present main findings and conclusions 

□ Perusal of students' work (dissertations etc.) 

□ Tour of the facilities 

□ Interviews with members of the institution 

□ Examination of documentary evidence 

□ Other (please specify) 

I ~H 

56. Who is interviewed at the site visit? 

□ Graduates 

□ Teaching staff 

□ Research staff 

□ Students 

□ Interviews are not used 

□ Management staff 

□ Administrative staff 

□ Other (please specify) 


9. Reporting the outcome of your Agency's external quality procedures 


57. Is there a publicly available report? 

O Yes 
O No 

Other (please specify) 

1 
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58. What is the content of the published report? 

□ Empirical evidence 

□ Analysis 

□ Conclusions 

□ Recommendations 

□ Conditions 

□ The report is not published 

□ Others (please specify) 

~H 

59. Information about the number of reports published by your Agency over the 
past three calendar years (i.e. January 2005-December 2007): 

How many institutional evaluation reports has your Agency 
published? 

How many thematic evaluation reports has your Agency 
published? 

How many subject evaluation reports has your Agency 
published? 

How many programme evaluation reports has your Agency 
published? 

60. Who is consulted before the report is published? 

□ The institution(s) that is(are) the subject of the external quality procedure 

□ Central government 

□ Regional government 

□ Industry and labour market 

□ Professional organisations 

□ National associations of HEIs (e.g. Rectors' Conference ...) 

□ Other (please specify) 

61. Who publishes the report? 

□ The quality assurance agency 

□ Central government 

□ Regional government 

□ The evaluated institution 

□ Others (please specify) 

HI 
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10. Follow-up 


62. Who is formally responsible for taking action/the following up on 
recommendations/conclusions of the report? 

□ The quality assurance agency 

□ There is no formal follow-up/action taken 

□ The institution that is the subject of the external quality procedure 

□ Central government 

□ Regional government 

□ Others (please specify) 

I *i 

63. What kind of actions are taken for the purpose of following up on 
recommendations/conclusions of the report? 

□ Desk-based study by quality assurance agency 

□ Follow-up site visit 

□ Request for action plan 

□ Repeat of external quality procedure 

□ Other (please specify) 

I i 

64. Is a systematic meta evaluation of the conducted external quality procedures 
carried out in your Agency's primary domain? 

O Yes 
O No 

Comment 

1 

If you answered 'No' to this question please skip to the next page. 
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65. Who carries out the systematic meta-evaluation of the conducted external 
quality procedures in your Agency's domain (process and report)? 

□ The quality assurance agency 

□ Central government 

□ Regional government 

□ An external agency 

□ There is no systematic meta-evaluation of the conducted evaluations 

□ Others (please specify) 

a 


11. Future Developments 


66. Please describe the expected developments in the QA sector in your Agency's 
primary domain in the next two years (i.e. by January 2010). 

□ Establishment of National Qualifications Framework 

□ Commencement of new higher education legislation 

□ Introduction of different methodologies for external quality procedures 

□ Revision of external quality procedures 

□ Reorganisation or merging of quality assurance agencies 

□ Other (please specify) 

I ~~3 

67. Please indicate which of the following assertions apply: 

□ My Agency has been externally reviewed. 

□ My Agency is not planning to subject itself to external review in the future 

□ My Agency plans to subject itself to external reviews in the future 

□ My Agency is currently preparing for being externally reviewed 

□ Other (please specify) 

I HI 

If your Agency is not planning to subject itself to external review in the future please skip the next question. 
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68. Please indicate how the review of your Agency has been/is/will be 
coordinated. 


Past evaluations 


Current evaluation 


Future evaluations 


Nationally coordinated 
Coordinated by ENQA 
Other (please specify) 


o 

o 


o 

o 


o 

o 


69. How do you regard the European Quality Assurance Register (EQAR) for 
Higher Education? 

□ Promoter of confidence between agencies 

□ Opportunity 

□ Useful tool to increase recognition 

□ Threat 

□ Other (please specify) 


1 


70. Is your Agency planning to apply for the Register? 

O No 
O Yes 

Other (please specify) 


1 


71. If your Agency plans to apply to be included on the Register what are the 
benefits that you expect to follow? 

□ Accountability 

□ Credibility 

□ National recognition as a quality assurance agency 

| | International recognition as a quality assurance agency 

□ National recognition as an accreditation agency 

□ International recognition as an accreditation agency 

□ Other (please specify) 


1 


12. Thank you 


Thank you for taking the time to complete this survey. 
Please click done to complete. 

You will then be redirected to the ENQA website. 






The present publication is the result of an ENQA survey on quality procedures 
of quality assurance agencies across Europe and beyond. This report updates the 
findings of the first ENQA survey on Quality Procedures in European Higher 
Education, undertaken by the Danish Evaluation Institute (EVA) in 2002, and 
considers the main developments that have taken place since then. 
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